2024-03-26T21:09:15.036 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T21:09:15.036 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} --archive /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129 --name gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi --owner scheduled_gabrioux@teuthology -- /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/orig.config.yaml 2024-03-26T21:09:15.070 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T21:09:15.159 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129 branch: wip-guits-testing-2024-03-26-0925 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7624129' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-26-0925 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2024-03-26-0925 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 832 sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-26-0925 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLl8JtZZia57JYJs08MTbvDulZpYGj2y1JE7EAG8GqkYw4GqBTxLBvOESKK+7Nsc2h2cBCEVEUnctAIu9B1fPCI= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPDWtrlkI5YzzjVmQcYHd88AMtVeANnyY2U5TB9zHZmkKGfAvcY6gUqtn3RyRUHvalZkcEGAT9kD7bFchZPqsjQ= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_20:31:35 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T21:09:15.159 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa; will attempt to use it 2024-03-26T21:09:15.160 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_42fc52b86b14d4f88c40d75eb7102e0ab3918f55/qa/tasks 2024-03-26T21:09:15.160 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T21:09:15.162 INFO:teuthology.task.internal:Checking packages... 2024-03-26T21:09:15.187 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '42fc52b86b14d4f88c40d75eb7102e0ab3918f55' 2024-03-26T21:09:15.187 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T21:09:15.187 INFO:teuthology.packaging:ref: None 2024-03-26T21:09:15.187 INFO:teuthology.packaging:tag: None 2024-03-26T21:09:15.187 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-26-0925 2024-03-26T21:09:15.187 INFO:teuthology.packaging:sha1: 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:09:15.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-26-0925 2024-03-26T21:09:15.504 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2484.g42fc52b8 2024-03-26T21:09:15.505 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T21:09:15.512 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T21:09:15.512 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T21:09:15.552 INFO:teuthology.task.internal:Saving configuration 2024-03-26T21:09:15.566 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T21:09:15.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T21:09:15.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:04:21.293410', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLl8JtZZia57JYJs08MTbvDulZpYGj2y1JE7EAG8GqkYw4GqBTxLBvOESKK+7Nsc2h2cBCEVEUnctAIu9B1fPCI='} 2024-03-26T21:09:15.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:04:21.291743', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPDWtrlkI5YzzjVmQcYHd88AMtVeANnyY2U5TB9zHZmkKGfAvcY6gUqtn3RyRUHvalZkcEGAT9kD7bFchZPqsjQ='} 2024-03-26T21:09:15.623 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T21:09:15.629 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T21:09:15.629 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T21:09:15.629 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T21:09:15.726 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9f697d2f70>, signals=[15]) 2024-03-26T21:09:15.727 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T21:09:15.736 INFO:teuthology.task.internal:Opening connections... 2024-03-26T21:09:15.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-03-26T21:09:15.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:09:15.857 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-03-26T21:09:15.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:09:15.961 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T21:09:15.967 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-03-26T21:09:15.985 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-03-26T21:09:15.985 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:09:16.043 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:09:16.044 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:09:16.044 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:09:16.044 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-26T21:09:16.076 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-03-26T21:09:16.093 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-03-26T21:09:16.094 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-03-26T21:09:16.150 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T21:09:16.151 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T21:09:16.151 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-03-26T21:09:16.184 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T21:09:16.193 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T21:09:16.249 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T21:09:16.249 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:09:16.251 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-03-26T21:09:16.267 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T21:09:16.283 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T21:09:16.283 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:09:16.310 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-03-26T21:09:16.339 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T21:09:16.393 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T21:09:16.407 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T21:09:16.408 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T21:09:16.408 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:09:16.409 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T21:09:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:09:16.425 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-26T21:09:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:09:16.426 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-26T21:09:16.482 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:09:16.482 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-361.el9.x86_64 2024-03-26T21:09:16.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-03-26T21:09:16.483 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-03-26T21:09:16.484 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-03-26T21:09:16.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-03-26T21:09:18.567 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T21:09:18.696 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T21:09:20.026 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-26T21:09:20.154 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-26T21:09:21.011 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T21:09:21.618 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-03-26T21:09:23.471 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-26T21:09:25.701 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-03-26T21:09:27.730 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-26T21:09:30.073 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-26T21:09:31.315 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-26T21:09:32.393 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-26T21:09:33.877 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-26T21:09:35.578 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-26T21:09:36.457 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-26T21:09:38.672 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:09:38.673 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:09:38.769 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:09:38.779 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:09:38.780 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-03-26T21:09:38.780 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:09:38.780 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 76 M 2024-03-26T21:09:38.780 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-03-26T21:09:38.780 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:09:39.034 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 496 kB/s | 66 kB 00:00 2024-03-26T21:09:39.484 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-03-26T21:09:40.284 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 24 MB/s | 33 MB 00:01 2024-03-26T21:09:40.678 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-26T21:09:41.210 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-26T21:09:41.210 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:09:41.210 INFO:teuthology.orchestra.run.smithi055.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-26T21:09:41.900 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:09:42.305 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:09:42.305 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:09:42.952 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T21:09:42.952 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T21:09:43.049 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:09:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:09:43.060 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-03-26T21:09:43.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:09:43.060 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 76 M 2024-03-26T21:09:43.060 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 123 M 2024-03-26T21:09:43.060 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:09:43.347 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:09:43.347 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:09:43.863 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 100 kB/s | 66 kB 00:00 2024-03-26T21:09:45.945 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:09:46.577 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:09:46.668 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:09:49.547 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:09:49.680 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:09:54.781 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:09:54.824 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:09:59.956 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:02.370 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 797 kB/s | 15 MB 00:19 2024-03-26T21:10:18.736 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 807 kB/s | 27 MB 00:34 2024-03-26T21:10:27.310 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 777 kB/s | 33 MB 00:44 2024-03-26T21:10:27.310 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:10:27.311 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.7 MB/s | 76 MB 00:44 2024-03-26T21:10:27.998 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:10:28.419 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:10:28.419 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:10:29.477 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:10:29.477 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:10:30.903 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:30.903 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:10:30.903 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:10:30.950 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:31.368 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:31.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:10:31.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:10:31.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:10:31.769 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:10:31.770 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:10:31.770 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:10:32.074 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:10:32.104 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:10:32.198 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:10:32.199 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:10:32.199 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:10:32.199 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:10:32.199 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:10:32.199 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T21:10:32.199 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:10:32.199 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:10:32.283 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:10:32.283 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:10:32.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-03-26T21:10:32.322 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:10:32.323 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:10:32.324 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:10:32.324 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:10:32.324 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-03-26T21:10:32.324 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:10:32.324 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:10:32.325 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:10:32.325 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:10:32.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:10:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:10:32.803 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:10:32.930 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-03-26T21:10:34.279 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:10:34.312 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-03-26T21:10:34.317 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-26T21:10:34.333 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.p6lQl8N8j0 2024-03-26T21:10:34.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p6lQl8N8j0 2024-03-26T21:10:34.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.p6lQl8N8j0 2024-03-26T21:10:34.549 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.p6lQl8N8j0 is 6KB 2024-03-26T21:10:34.597 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.p6lQl8N8j0 2024-03-26T21:10:34.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:10:34.682 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:10:34.683 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:10:34.683 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:10:34.683 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:10:34.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:10:34.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-03-26T21:10:35.670 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:10:35.812 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:10:40.933 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:10:41.153 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:10:46.282 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:11:04.825 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T21:11:04.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:11:04.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:11:17.170 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:11:17.170 INFO:teuthology.orchestra.run.smithi067.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T21:11:17.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:11:17.216 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:11:17.663 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:11:17.663 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T21:11:17.663 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T21:11:17.663 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T21:11:18.075 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T21:11:18.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:11:18.076 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:11:18.415 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:11:18.562 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.562 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:11:18.562 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T21:11:18.562 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T21:11:18.562 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T21:11:18.562 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T21:11:18.562 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T21:11:18.563 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T21:11:18.645 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T21:11:18.645 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T21:11:18.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T21:11:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T21:11:18.687 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T21:11:18.688 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-03-26T21:11:18.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:11:18.688 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T21:11:18.688 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T21:11:18.688 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-432.el9.x86_64 2024-03-26T21:11:18.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T21:11:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-03-26T21:11:20.866 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T21:11:20.901 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-03-26T21:11:20.903 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-26T21:11:20.918 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.uP7wffJ9ZK 2024-03-26T21:11:20.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uP7wffJ9ZK 2024-03-26T21:11:20.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.uP7wffJ9ZK 2024-03-26T21:11:21.271 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.uP7wffJ9ZK is 6KB 2024-03-26T21:11:21.320 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.uP7wffJ9ZK 2024-03-26T21:11:21.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T21:11:21.413 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T21:11:21.414 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T21:11:21.414 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T21:11:21.414 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T21:11:21.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T21:11:21.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-03-26T21:11:23.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-26T21:11:32.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:11:32.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:11:51.590 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T21:11:51.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:11:51.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:10.019 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-26T21:12:19.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:12:19.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:22.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-26T21:12:32.190 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T21:12:34.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:12:34.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:37.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-26T21:12:44.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:12:44.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:44.554 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-26T21:12:44.929 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:12:44.929 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:12:44.929 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-26T21:12:44.944 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:12:44.944 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:12:44.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:12:44.945 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:12:45.946 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T21:12:45.946 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:12:46.024 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-03-26T21:12:46.064 DEBUG:teuthology.parallel:result is None 2024-03-26T21:12:52.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:12:52.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:12:55.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-26T21:13:13.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:13:13.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:13:13.588 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-26T21:13:14.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:13:14.012 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T21:13:14.013 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-26T21:13:14.028 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-432.el9.x86_64 2024-03-26T21:13:14.028 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T21:13:14.028 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T21:13:14.028 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T21:13:15.030 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T21:13:15.030 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T21:13:15.111 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-03-26T21:13:15.159 DEBUG:teuthology.parallel:result is None 2024-03-26T21:13:15.159 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T21:13:15.167 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T21:13:15.167 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:13:15.169 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T21:13:15.186 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T21:13:15.198 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T21:13:15.207 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T21:13:15.207 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:13:15.227 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T21:13:15.257 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T21:13:15.265 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T21:13:15.266 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:13:15.287 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T21:13:15.321 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:13:15.332 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:13:15.335 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:13:15.345 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T21:13:15.347 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T21:13:15.354 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T21:13:15.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:13:15.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T21:13:15.416 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T21:13:15.425 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T21:13:15.425 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:13:15.449 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T21:13:15.472 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:13:15.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:13:15.592 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:13:15.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:13:15.718 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:13:15.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:13:15.786 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:13:15.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T21:13:15.884 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:13:15.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T21:13:16.013 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:13:16.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T21:13:16.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-03-26T21:13:16.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-03-26T21:13:16.125 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:13:16.173 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T21:13:16.607 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T21:13:16.631 INFO:teuthology.task.internal:Starting timer... 2024-03-26T21:13:16.631 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T21:13:16.642 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T21:13:16.675 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T21:13:16.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-03-26T21:13:16.736 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-03-26T21:13:16.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-03-26T21:13:16.808 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-03-26T21:13:16.810 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T21:13:16.810 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-03-26T21:13:16.846 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-03-26T21:13:16.846 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-03-26T21:13:16.870 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-03-26T21:13:16.870 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2024-03-26T21:13:16.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T21:13:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:13:16.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T21:13:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:13:16.947 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T21:13:16.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:13:16.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-03-26T21:13:17.030 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T21:13:17.040 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T21:13:17.056 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T21:13:17.057 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T21:19:38.269 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2024-03-26T21:19:38.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:19:38.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:19:38.362 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-26T21:19:38.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-26T21:19:38.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:19:38.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T21:19:38.533 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-26T21:19:38.608 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-26T21:19:38.609 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T21:19:38.619 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T21:19:38.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:19:38.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:19:38.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T21:19:38.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T21:19:38.651 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:19:38.667 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:19:38.680 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T21:19:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T21:19:38.697 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-03-26T21:19:38.711 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-03-26T21:19:38.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-03-26T21:19:38.727 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:19:38.740 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-03-26T21:19:38.743 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:19:38.756 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T21:19:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T21:19:38.796 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.800 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-26T21:19:38.831 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T21:19:38.832 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-26T21:19:38.832 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.832 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.832 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.832 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T21:19:38.832 INFO:teuthology.run_tasks:Running task install... 2024-03-26T21:19:38.842 DEBUG:teuthology.task.install:project ceph 2024-03-26T21:19:38.842 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}} 2024-03-26T21:19:38.842 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'} 2024-03-26T21:19:38.842 INFO:teuthology.task.install:Using flavor: default 2024-03-26T21:19:38.847 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T21:19:38.848 INFO:teuthology.task.install:extra packages: [] 2024-03-26T21:19:38.848 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:19:38.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:19:38.850 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'tag': None, 'wait_for_package': False} 2024-03-26T21:19:38.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:19:38.997 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/ 2024-03-26T21:19:38.997 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2484.g42fc52b8 2024-03-26T21:19:39.107 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:19:39.107 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:19:39.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:19:39.136 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:19:39.136 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:19:39.147 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/ 2024-03-26T21:19:39.148 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2484.g42fc52b8 2024-03-26T21:19:39.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:19:39.287 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:19:39.353 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-03-26T21:19:39.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-03-26T21:19:39.361 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T21:19:39.361 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:19:39.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T21:19:39.390 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T21:19:39.390 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T21:19:39.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T21:19:39.539 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T21:19:39.600 INFO:teuthology.orchestra.run.smithi055.stdout:74 files removed 2024-03-26T21:19:39.605 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-03-26T21:19:39.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-03-26T21:19:39.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2024-03-26T21:19:39.871 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-03-26T21:19:39.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-03-26T21:19:40.423 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 170 kB/s | 83 kB 00:00 2024-03-26T21:19:40.626 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 196 kB/s | 83 kB 00:00 2024-03-26T21:19:40.966 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-03-26T21:19:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-03-26T21:19:41.333 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-26T21:19:41.377 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-26T21:19:41.896 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T21:19:41.902 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T21:19:43.364 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-26T21:19:43.486 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-26T21:19:52.210 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 983 kB/s | 8.0 MB 00:08 2024-03-26T21:19:52.670 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 919 kB/s | 8.0 MB 00:08 2024-03-26T21:19:54.177 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-26T21:19:55.538 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-26T21:19:58.329 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-26T21:19:59.764 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-26T21:20:00.154 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-26T21:20:01.356 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-26T21:20:02.181 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-26T21:20:04.849 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-26T21:20:06.398 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-26T21:20:08.945 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:20:08.957 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:08.957 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:20:08.957 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:08.957 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:20:08.957 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 9.8 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.4 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.4 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 21 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 25 k 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:20:08.958 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 746 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 500 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.1 M 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 45 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 134 k 2024-03-26T21:20:08.959 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 162 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 321 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 298 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 99 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout:Install 30 Packages 2024-03-26T21:20:08.960 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2024-03-26T21:20:08.961 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:08.961 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 59 M 2024-03-26T21:20:08.961 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:20:10.033 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T21:20:10.391 INFO:teuthology.orchestra.run.smithi055.stdout:(1/32): ceph-radosgw-19.0.0-2484.g42fc52b8.el9. 12 MB/s | 9.8 MB 00:00 2024-03-26T21:20:10.441 INFO:teuthology.orchestra.run.smithi055.stdout:(2/32): ceph-selinux-19.0.0-2484.g42fc52b8.el9. 507 kB/s | 25 kB 00:00 2024-03-26T21:20:10.533 INFO:teuthology.orchestra.run.smithi055.stdout:(3/32): libcephfs2-19.0.0-2484.g42fc52b8.el9.x8 8.0 MB/s | 746 kB 00:00 2024-03-26T21:20:10.600 INFO:teuthology.orchestra.run.smithi055.stdout:(4/32): libradosstriper1-19.0.0-2484.g42fc52b8. 7.3 MB/s | 500 kB 00:00 2024-03-26T21:20:10.851 INFO:teuthology.orchestra.run.smithi055.stdout:(5/32): ceph-base-19.0.0-2484.g42fc52b8.el9.x86 4.3 MB/s | 5.4 MB 00:01 2024-03-26T21:20:10.926 INFO:teuthology.orchestra.run.smithi055.stdout:(6/32): librgw2-19.0.0-2484.g42fc52b8.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-26T21:20:10.951 INFO:teuthology.orchestra.run.smithi055.stdout:(7/32): python3-ceph-argparse-19.0.0-2484.g42fc 451 kB/s | 45 kB 00:00 2024-03-26T21:20:10.976 INFO:teuthology.orchestra.run.smithi055.stdout:(8/32): python3-ceph-common-19.0.0-2484.g42fc52 2.6 MB/s | 134 kB 00:00 2024-03-26T21:20:11.010 INFO:teuthology.orchestra.run.smithi055.stdout:(9/32): python3-cephfs-19.0.0-2484.g42fc52b8.el 2.7 MB/s | 162 kB 00:00 2024-03-26T21:20:11.035 INFO:teuthology.orchestra.run.smithi055.stdout:(10/32): python3-rados-19.0.0-2484.g42fc52b8.el 5.4 MB/s | 321 kB 00:00 2024-03-26T21:20:11.077 INFO:teuthology.orchestra.run.smithi055.stdout:(11/32): python3-rbd-19.0.0-2484.g42fc52b8.el9. 4.3 MB/s | 298 kB 00:00 2024-03-26T21:20:11.094 INFO:teuthology.orchestra.run.smithi055.stdout:(12/32): python3-rgw-19.0.0-2484.g42fc52b8.el9. 1.7 MB/s | 99 kB 00:00 2024-03-26T21:20:11.244 INFO:teuthology.orchestra.run.smithi055.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi055.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-26T21:20:11.287 INFO:teuthology.orchestra.run.smithi055.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T21:20:11.354 INFO:teuthology.orchestra.run.smithi055.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-26T21:20:11.388 INFO:teuthology.orchestra.run.smithi055.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-26T21:20:11.413 INFO:teuthology.orchestra.run.smithi055.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2024-03-26T21:20:11.505 INFO:teuthology.orchestra.run.smithi055.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 391 kB/s | 36 kB 00:00 2024-03-26T21:20:11.555 INFO:teuthology.orchestra.run.smithi055.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.3 MB/s | 221 kB 00:00 2024-03-26T21:20:11.580 INFO:teuthology.orchestra.run.smithi055.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-03-26T21:20:12.114 INFO:teuthology.orchestra.run.smithi055.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 553 kB/s | 309 kB 00:00 2024-03-26T21:20:12.240 INFO:teuthology.orchestra.run.smithi055.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-26T21:20:12.540 INFO:teuthology.orchestra.run.smithi055.stdout:(24/32): ceph-common-19.0.0-2484.g42fc52b8.el9. 7.0 MB/s | 21 MB 00:02 2024-03-26T21:20:12.565 INFO:teuthology.orchestra.run.smithi055.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 149 kB/s | 49 kB 00:00 2024-03-26T21:20:12.578 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 9.8 M 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-03-26T21:20:12.591 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.4 M 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.4 M 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 21 M 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 25 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 746 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T21:20:12.592 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 500 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 5.1 M 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 45 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 134 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 162 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 321 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 298 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 99 k 2024-03-26T21:20:12.593 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout:Install 30 Packages 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-03-26T21:20:12.594 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:12.595 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 59 M 2024-03-26T21:20:12.595 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:20:12.641 INFO:teuthology.orchestra.run.smithi055.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-03-26T21:20:12.749 INFO:teuthology.orchestra.run.smithi055.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-26T21:20:12.891 INFO:teuthology.orchestra.run.smithi055.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-26T21:20:12.916 INFO:teuthology.orchestra.run.smithi055.stdout:(29/32): libunwind-1.6.2-1.el9.x86_64.rpm 179 kB/s | 67 kB 00:00 2024-03-26T21:20:12.967 INFO:teuthology.orchestra.run.smithi055.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-26T21:20:13.267 INFO:teuthology.orchestra.run.smithi055.stdout:(31/32): librbd1-19.0.0-2484.g42fc52b8.el9.x86_ 9.0 MB/s | 3.1 MB 00:00 2024-03-26T21:20:13.726 INFO:teuthology.orchestra.run.smithi055.stdout:(32/32): librados2-19.0.0-2484.g42fc52b8.el9.x8 4.1 MB/s | 3.4 MB 00:00 2024-03-26T21:20:13.729 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:13.729 INFO:teuthology.orchestra.run.smithi055.stdout:Total 12 MB/s | 59 MB 00:04 2024-03-26T21:20:13.763 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:20:13.804 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:20:13.804 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:20:14.310 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:20:14.310 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:20:14.442 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-common-19.0.0-2484.g42fc52b8.el9.x 13 MB/s | 21 MB 00:01 2024-03-26T21:20:14.484 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-selinux-19.0.0-2484.g42fc52b8.el9. 610 kB/s | 25 kB 00:00 2024-03-26T21:20:14.551 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): libcephfs2-19.0.0-2484.g42fc52b8.el9.x8 11 MB/s | 746 kB 00:00 2024-03-26T21:20:14.610 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): libradosstriper1-19.0.0-2484.g42fc52b8. 8.4 MB/s | 500 kB 00:00 2024-03-26T21:20:14.902 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): librgw2-19.0.0-2484.g42fc52b8.el9.x86_6 18 MB/s | 5.1 MB 00:00 2024-03-26T21:20:14.944 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): python3-ceph-argparse-19.0.0-2484.g42fc 1.1 MB/s | 45 kB 00:00 2024-03-26T21:20:14.994 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): python3-ceph-common-19.0.0-2484.g42fc52 2.6 MB/s | 134 kB 00:00 2024-03-26T21:20:15.044 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-cephfs-19.0.0-2484.g42fc52b8.el 3.2 MB/s | 162 kB 00:00 2024-03-26T21:20:15.095 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): python3-rados-19.0.0-2484.g42fc52b8.el9 6.3 MB/s | 321 kB 00:00 2024-03-26T21:20:15.145 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): python3-rbd-19.0.0-2484.g42fc52b8.el9. 5.8 MB/s | 298 kB 00:00 2024-03-26T21:20:15.195 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): python3-rgw-19.0.0-2484.g42fc52b8.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T21:20:15.329 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 794 kB/s | 106 kB 00:00 2024-03-26T21:20:15.388 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-26T21:20:15.429 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T21:20:15.505 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T21:20:15.547 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-26T21:20:15.589 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-26T21:20:15.755 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.3 MB/s | 221 kB 00:00 2024-03-26T21:20:15.797 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 862 kB/s | 36 kB 00:00 2024-03-26T21:20:15.811 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:20:15.839 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-03-26T21:20:15.900 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T21:20:15.931 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T21:20:15.956 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T21:20:15.979 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:20:15.995 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:20:16.073 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-26T21:20:16.146 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:20:16.202 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:20:16.507 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-26T21:20:16.557 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-03-26T21:20:16.560 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T21:20:16.607 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-26T21:20:16.633 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:16.658 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-26T21:20:16.733 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-26T21:20:16.745 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:16.783 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-26T21:20:16.874 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 8/34 2024-03-26T21:20:16.902 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:20:16.909 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-26T21:20:16.964 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:20:17.002 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T21:20:17.046 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 11/34 2024-03-26T21:20:17.116 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:20:17.126 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): ceph-base-19.0.0-2484.g42fc52b8.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-03-26T21:20:17.137 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:20:17.185 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:20:17.193 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): librados2-19.0.0-2484.g42fc52b8.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-26T21:20:17.220 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T21:20:17.332 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T21:20:17.364 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T21:20:17.404 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T21:20:17.436 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T21:20:17.465 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T21:20:17.498 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T21:20:17.844 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T21:20:17.869 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:20:17.944 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:20:17.974 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 23/34 2024-03-26T21:20:17.978 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): ceph-radosgw-19.0.0-2484.g42fc52b8.el9 1.9 MB/s | 9.8 MB 00:05 2024-03-26T21:20:18.020 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T21:20:18.477 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): librbd1-19.0.0-2484.g42fc52b8.el9.x86_ 2.3 MB/s | 3.1 MB 00:01 2024-03-26T21:20:18.481 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:18.481 INFO:teuthology.orchestra.run.smithi067.stdout:Total 10 MB/s | 59 MB 00:05 2024-03-26T21:20:18.504 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T21:20:18.514 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:20:18.531 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:20:18.558 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:20:18.558 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:20:18.586 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:20:18.656 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 27/34 2024-03-26T21:20:18.700 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 28/34 2024-03-26T21:20:19.052 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:20:19.053 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:20:20.533 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:20:20.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T21:20:20.648 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T21:20:20.674 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T21:20:20.696 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:20:20.713 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T21:20:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:20:20.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T21:20:21.128 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:21.164 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:21.306 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T21:20:21.358 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:21.451 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 8/34 2024-03-26T21:20:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:20:21.639 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 9/34 2024-03-26T21:20:21.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T21:20:21.695 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 11/34 2024-03-26T21:20:21.747 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:21.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:20:21.785 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:20:21.787 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:20:21.851 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 13/34 2024-03-26T21:20:21.886 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T21:20:21.898 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:20:21.898 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:20:21.898 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:20:21.899 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:21.944 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T21:20:22.037 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T21:20:22.077 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T21:20:22.110 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T21:20:22.139 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T21:20:22.171 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T21:20:22.506 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T21:20:22.534 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:20:22.625 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 22/34 2024-03-26T21:20:22.648 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 23/34 2024-03-26T21:20:22.689 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T21:20:23.176 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T21:20:23.204 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:20:23.267 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 26/34 2024-03-26T21:20:23.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 27/34 2024-03-26T21:20:23.373 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 28/34 2024-03-26T21:20:25.634 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:26.385 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 29/34 2024-03-26T21:20:26.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:20:26.523 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 30/34 2024-03-26T21:20:26.524 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T21:20:26.524 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T21:20:26.524 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:26.557 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-03-26T21:20:32.281 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:33.253 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:20:33.277 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:33.279 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:20:33.353 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:20:33.353 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:34.504 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 6/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 8/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 9/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 10/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 11/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:20:34.505 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 13/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T21:20:34.506 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T21:20:34.507 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T21:20:34.507 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T21:20:34.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:34.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T21:20:34.508 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/34 2024-03-26T21:20:35.553 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.554 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.555 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:35.556 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:20:35.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2024-03-26T21:20:36.359 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:30 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-03-26T21:20:36.474 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-03-26T21:20:36.475 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-03-26T21:20:36.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:36.514 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:20:36.515 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:36.515 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:20:36.515 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:36.515 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:20:36.515 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 48 M 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 163 k 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-26T21:20:36.516 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:36.517 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 49 M 2024-03-26T21:20:36.517 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 206 M 2024-03-26T21:20:36.517 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:20:36.841 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-26T21:20:36.899 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T21:20:36.941 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-26T21:20:37.025 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-26T21:20:37.058 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): libcephsqlite-19.0.0-2484.g42fc52b8.el9. 402 kB/s | 163 kB 00:00 2024-03-26T21:20:37.437 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:20:37.461 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/34 2024-03-26T21:20:37.461 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:37.461 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T21:20:37.462 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:20:37.462 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T21:20:37.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:37.463 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:20:37.558 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T21:20:37.558 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_6 6/34 2024-03-26T21:20:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 7/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9. 8/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x8 9/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 10/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 11/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 12/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 13/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T21:20:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 31/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T21:20:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/34 2024-03-26T21:20:39.159 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-19.0.0-2484.g42fc52b8.el9.x86_ 19 MB/s | 48 MB 00:02 2024-03-26T21:20:39.162 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:39.163 INFO:teuthology.orchestra.run.smithi055.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-26T21:20:39.177 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:20:39.199 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:20:39.200 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:20:39.606 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:20:39.606 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:20:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T21:20:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:39.754 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:20:40.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-03-26T21:20:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:20:40.275 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:20:40.322 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:20:40.358 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:20:40.402 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:20:40.428 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:20:40.563 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:30 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:20:40.718 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 48 M 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 163 k 2024-03-26T21:20:40.719 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-03-26T21:20:40.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:40.721 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 49 M 2024-03-26T21:20:40.721 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 206 M 2024-03-26T21:20:40.721 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:20:41.220 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 681 kB/s | 217 kB 00:00 2024-03-26T21:20:41.278 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): libcephsqlite-19.0.0-2484.g42fc52b8.el9. 433 kB/s | 163 kB 00:00 2024-03-26T21:20:41.320 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-26T21:20:41.345 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 936 kB/s | 63 kB 00:00 2024-03-26T21:20:41.412 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-26T21:20:43.563 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-19.0.0-2484.g42fc52b8.el9.x86_ 18 MB/s | 48 MB 00:02 2024-03-26T21:20:43.566 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:43.566 INFO:teuthology.orchestra.run.smithi067.stdout:Total 17 MB/s | 49 MB 00:02 2024-03-26T21:20:43.581 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:20:43.603 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:20:43.604 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:20:44.007 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:20:44.008 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:20:44.598 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:20:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T21:20:44.736 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T21:20:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:20:44.813 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T21:20:44.841 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:20:45.190 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:20:45.263 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:20:47.558 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:20:47.558 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/6 2024-03-26T21:20:47.559 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/6 2024-03-26T21:20:47.559 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:20:47.559 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:20:47.559 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:20:47.898 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:47.899 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:20:47.899 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:20:47.899 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:20:47.899 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:47.899 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:20:48.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2024-03-26T21:20:48.789 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:42 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:20:48.950 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:20:48.952 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 6.6 k 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 2.4 M 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 1.5 M 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 249 k 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 4.7 M 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 17 M 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T21:20:48.953 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:20:48.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:20:48.955 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout:Install 36 Packages 2024-03-26T21:20:48.956 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:48.957 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 30 M 2024-03-26T21:20:48.957 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 108 M 2024-03-26T21:20:48.957 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:20:49.562 INFO:teuthology.orchestra.run.smithi055.stdout:(1/36): ceph-19.0.0-2484.g42fc52b8.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-03-26T21:20:49.604 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/6 2024-03-26T21:20:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:20:49.771 INFO:teuthology.orchestra.run.smithi055.stdout:(2/36): ceph-mds-19.0.0-2484.g42fc52b8.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-03-26T21:20:50.079 INFO:teuthology.orchestra.run.smithi055.stdout:(3/36): ceph-mgr-19.0.0-2484.g42fc52b8.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-03-26T21:20:50.146 INFO:teuthology.orchestra.run.smithi055.stdout:(4/36): ceph-mon-19.0.0-2484.g42fc52b8.el9.x86_ 8.0 MB/s | 4.7 MB 00:00 2024-03-26T21:20:50.180 INFO:teuthology.orchestra.run.smithi055.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2484.g42fc 2.4 MB/s | 249 kB 00:00 2024-03-26T21:20:50.330 INFO:teuthology.orchestra.run.smithi055.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-26T21:20:50.355 INFO:teuthology.orchestra.run.smithi055.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 956 kB/s | 144 kB 00:00 2024-03-26T21:20:50.381 INFO:teuthology.orchestra.run.smithi055.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-26T21:20:50.406 INFO:teuthology.orchestra.run.smithi055.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-26T21:20:50.431 INFO:teuthology.orchestra.run.smithi055.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-26T21:20:50.456 INFO:teuthology.orchestra.run.smithi055.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T21:20:50.523 INFO:teuthology.orchestra.run.smithi055.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-03-26T21:20:50.548 INFO:teuthology.orchestra.run.smithi055.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-26T21:20:50.574 INFO:teuthology.orchestra.run.smithi055.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T21:20:50.599 INFO:teuthology.orchestra.run.smithi055.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-26T21:20:50.766 INFO:teuthology.orchestra.run.smithi055.stdout:(16/36): ceph-osd-19.0.0-2484.g42fc52b8.el9.x86 17 MB/s | 17 MB 00:00 2024-03-26T21:20:50.900 INFO:teuthology.orchestra.run.smithi055.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-26T21:20:50.950 INFO:teuthology.orchestra.run.smithi055.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-03-26T21:20:51.067 INFO:teuthology.orchestra.run.smithi055.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2024-03-26T21:20:51.209 INFO:teuthology.orchestra.run.smithi055.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 197 kB/s | 125 kB 00:00 2024-03-26T21:20:51.268 INFO:teuthology.orchestra.run.smithi055.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-26T21:20:51.326 INFO:teuthology.orchestra.run.smithi055.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-26T21:20:51.393 INFO:teuthology.orchestra.run.smithi055.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 142 kB/s | 18 kB 00:00 2024-03-26T21:20:51.444 INFO:teuthology.orchestra.run.smithi055.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 366 kB/s | 172 kB 00:00 2024-03-26T21:20:51.543 INFO:teuthology.orchestra.run.smithi055.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-26T21:20:51.594 INFO:teuthology.orchestra.run.smithi055.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-26T21:20:51.686 INFO:teuthology.orchestra.run.smithi055.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 326 kB/s | 46 kB 00:00 2024-03-26T21:20:51.703 INFO:teuthology.orchestra.run.smithi055.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 84 kB/s | 20 kB 00:00 2024-03-26T21:20:51.811 INFO:teuthology.orchestra.run.smithi055.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 153 kB/s | 16 kB 00:00 2024-03-26T21:20:51.920 INFO:teuthology.orchestra.run.smithi055.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 243 kB/s | 79 kB 00:00 2024-03-26T21:20:51.945 INFO:teuthology.orchestra.run.smithi055.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 678 kB/s | 90 kB 00:00 2024-03-26T21:20:52.062 INFO:teuthology.orchestra.run.smithi055.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 252 kB/s | 36 kB 00:00 2024-03-26T21:20:52.121 INFO:teuthology.orchestra.run.smithi055.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-26T21:20:52.175 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 6/6 2024-03-26T21:20:52.175 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/6 2024-03-26T21:20:52.176 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/6 2024-03-26T21:20:52.176 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T21:20:52.176 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T21:20:52.176 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T21:20:52.178 INFO:teuthology.orchestra.run.smithi055.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 554 kB/s | 272 kB 00:00 2024-03-26T21:20:52.296 INFO:teuthology.orchestra.run.smithi055.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-26T21:20:52.438 INFO:teuthology.orchestra.run.smithi055.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 63 kB/s | 20 kB 00:00 2024-03-26T21:20:52.444 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:52.444 INFO:teuthology.orchestra.run.smithi055.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-03-26T21:20:52.467 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:20:52.509 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:20:52.509 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:20:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T21:20:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:52.604 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:52.605 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:20:52.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-03-26T21:20:52.932 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:20:52.932 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:20:53.484 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:43 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:20:53.643 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:20:53.645 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-26T21:20:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 6.6 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 2.4 M 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 1.5 M 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 249 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 4.7 M 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 17 M 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T21:20:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T21:20:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-26T21:20:53.648 INFO:teuthology.orchestra.run.smithi067.stdout:Install 36 Packages 2024-03-26T21:20:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:20:53.649 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 30 M 2024-03-26T21:20:53.649 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 108 M 2024-03-26T21:20:53.649 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:20:53.927 INFO:teuthology.orchestra.run.smithi067.stdout:(1/36): ceph-19.0.0-2484.g42fc52b8.el9.x86_64.r 44 kB/s | 6.6 kB 00:00 2024-03-26T21:20:54.244 INFO:teuthology.orchestra.run.smithi067.stdout:(2/36): ceph-mds-19.0.0-2484.g42fc52b8.el9.x86_ 5.1 MB/s | 2.4 MB 00:00 2024-03-26T21:20:54.268 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:20:54.333 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T21:20:54.368 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T21:20:54.394 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T21:20:54.460 INFO:teuthology.orchestra.run.smithi067.stdout:(3/36): ceph-mon-19.0.0-2484.g42fc52b8.el9.x86_ 8.8 MB/s | 4.7 MB 00:00 2024-03-26T21:20:54.461 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T21:20:54.510 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T21:20:54.510 INFO:teuthology.orchestra.run.smithi067.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2484.g42fc 4.9 MB/s | 249 kB 00:00 2024-03-26T21:20:54.542 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T21:20:54.640 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T21:20:54.660 INFO:teuthology.orchestra.run.smithi067.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-26T21:20:54.711 INFO:teuthology.orchestra.run.smithi067.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-26T21:20:54.725 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T21:20:54.753 INFO:teuthology.orchestra.run.smithi067.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 761 kB/s | 32 kB 00:00 2024-03-26T21:20:54.771 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T21:20:54.795 INFO:teuthology.orchestra.run.smithi067.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-26T21:20:54.837 INFO:teuthology.orchestra.run.smithi067.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2024-03-26T21:20:54.899 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T21:20:54.899 INFO:teuthology.orchestra.run.smithi067.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T21:20:54.906 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T21:20:54.951 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T21:20:54.970 INFO:teuthology.orchestra.run.smithi067.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T21:20:55.010 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T21:20:55.012 INFO:teuthology.orchestra.run.smithi067.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-26T21:20:55.054 INFO:teuthology.orchestra.run.smithi067.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-26T21:20:55.076 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T21:20:55.088 INFO:teuthology.orchestra.run.smithi067.stdout:(14/36): ceph-mgr-19.0.0-2484.g42fc52b8.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-03-26T21:20:55.113 INFO:teuthology.orchestra.run.smithi067.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-26T21:20:55.280 INFO:teuthology.orchestra.run.smithi067.stdout:(16/36): ceph-osd-19.0.0-2484.g42fc52b8.el9.x86 16 MB/s | 17 MB 00:01 2024-03-26T21:20:55.285 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T21:20:55.338 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T21:20:55.380 INFO:teuthology.orchestra.run.smithi067.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 428 kB/s | 125 kB 00:00 2024-03-26T21:20:55.383 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T21:20:55.420 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T21:20:55.450 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T21:20:55.531 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T21:20:55.568 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:20:55.602 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T21:20:55.642 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T21:20:55.687 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T21:20:55.706 INFO:teuthology.orchestra.run.smithi067.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2024-03-26T21:20:55.730 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T21:20:55.764 INFO:teuthology.orchestra.run.smithi067.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-03-26T21:20:55.825 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T21:20:55.859 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T21:20:55.937 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T21:20:55.965 INFO:teuthology.orchestra.run.smithi067.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 26 kB/s | 22 kB 00:00 2024-03-26T21:20:55.990 INFO:teuthology.orchestra.run.smithi067.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 605 kB/s | 172 kB 00:00 2024-03-26T21:20:56.015 INFO:teuthology.orchestra.run.smithi067.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-26T21:20:56.090 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T21:20:56.099 INFO:teuthology.orchestra.run.smithi067.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-26T21:20:56.116 INFO:teuthology.orchestra.run.smithi067.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-26T21:20:56.133 INFO:teuthology.orchestra.run.smithi067.stdout:(25/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-03-26T21:20:56.192 INFO:teuthology.orchestra.run.smithi067.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-26T21:20:56.208 INFO:teuthology.orchestra.run.smithi067.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-26T21:20:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:20:56.317 INFO:teuthology.orchestra.run.smithi067.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 630 kB/s | 79 kB 00:00 2024-03-26T21:20:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 231 kB/s | 46 kB 00:00 2024-03-26T21:20:56.359 INFO:teuthology.orchestra.run.smithi067.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-26T21:20:56.409 INFO:teuthology.orchestra.run.smithi067.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-26T21:20:56.451 INFO:teuthology.orchestra.run.smithi067.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-26T21:20:56.526 INFO:teuthology.orchestra.run.smithi067.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 468 kB/s | 90 kB 00:00 2024-03-26T21:20:56.569 INFO:teuthology.orchestra.run.smithi067.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-26T21:20:56.586 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 31/36 2024-03-26T21:20:56.610 INFO:teuthology.orchestra.run.smithi067.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-26T21:20:56.622 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:20:56.627 INFO:teuthology.orchestra.run.smithi067.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-26T21:20:56.631 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:20:56.631 INFO:teuthology.orchestra.run.smithi067.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:20:56.648 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:56.656 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:20:56.703 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:20:56.704 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:20:57.119 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:20:57.119 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:20:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:20:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:20:58.190 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:58.190 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:20:58.191 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:20:58.191 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:20:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:58.451 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:20:58.513 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T21:20:58.548 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T21:20:58.582 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T21:20:58.656 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T21:20:58.657 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:20:58.680 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:58.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T21:20:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T21:20:58.826 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T21:20:58.904 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T21:20:58.924 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:20:58.947 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:20:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T21:20:59.009 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:20:59.024 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T21:20:59.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T21:20:59.114 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T21:20:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T21:20:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T21:20:59.437 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T21:20:59.484 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T21:20:59.538 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T21:20:59.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T21:20:59.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T21:20:59.676 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T21:20:59.715 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:20:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T21:20:59.788 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T21:20:59.826 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T21:20:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T21:20:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T21:20:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T21:21:00.064 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T21:21:00.213 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T21:21:00.343 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:21:00.513 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 31/36 2024-03-26T21:21:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 32/36 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T21:21:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:00.785 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 6/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T21:21:00.786 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T21:21:00.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T21:21:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T21:21:00.788 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T21:21:00.789 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T21:21:01.905 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T21:21:01.906 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:21:01.907 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:01.908 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:21:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 33/36 2024-03-26T21:21:02.119 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:02.120 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T21:21:02.120 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:21:02.120 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T21:21:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:02.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2024-03-26T21:21:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:21:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 34/36 2024-03-26T21:21:02.609 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:02.609 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T21:21:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:21:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T21:21:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:02.821 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:02.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 35/36 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T21:21:02.860 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:21:02.920 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:02.976 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:02.977 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:02.977 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:03.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2024-03-26T21:21:03.606 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:03.761 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 762 k 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:03.762 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:03.763 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 762 k 2024-03-26T21:21:03.763 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 768 k 2024-03-26T21:21:03.763 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:04.549 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-19.0.0-2484.g42fc52b8.el9.noarch.rpm 969 kB/s | 762 kB 00:00 2024-03-26T21:21:04.550 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:04.551 INFO:teuthology.orchestra.run.smithi055.stdout:Total 967 kB/s | 762 kB 00:00 2024-03-26T21:21:04.551 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:04.557 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:04.557 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:04.563 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:04.563 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 36/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 3/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 4/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 5/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9. 6/36 2024-03-26T21:21:04.659 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T21:21:04.660 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T21:21:04.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T21:21:04.759 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:05.221 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:05.323 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T21:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T21:21:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T21:21:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T21:21:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:05.691 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:05.716 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:06.007 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:06.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-03-26T21:21:06.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:21:06.635 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:06.734 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:06.789 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:06.790 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:06.790 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:06.816 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:06.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-03-26T21:21:06.971 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:06.972 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:06.972 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-26T21:21:06.972 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:06.972 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:06.972 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 145 k 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 145 k 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 435 k 2024-03-26T21:21:06.973 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:07.418 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:07.573 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-19.0.0-2484.g42fc52 242 kB/s | 145 kB 00:00 2024-03-26T21:21:07.573 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi055.stdout:Total 241 kB/s | 145 kB 00:00 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 762 k 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:07.574 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:07.575 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 762 k 2024-03-26T21:21:07.575 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 768 k 2024-03-26T21:21:07.575 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:07.582 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:07.582 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:07.639 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:07.639 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:07.794 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:07.856 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:07.881 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:07.881 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:07.881 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:21:07.881 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:07.900 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-19.0.0-2484.g42fc52b8.el9.noarch.rpm 2.3 MB/s | 762 kB 00:00 2024-03-26T21:21:07.900 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:07.901 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.3 MB/s | 762 kB 00:00 2024-03-26T21:21:07.901 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:07.906 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:07.906 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:07.912 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:07.912 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:07.989 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:08.334 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:08.869 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:08.927 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:09.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2024-03-26T21:21:09.212 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/1 2024-03-26T21:21:09.212 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:09.213 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:09.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:09.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T21:21:09.709 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:09.808 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:09.863 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:09.864 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:09.864 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:09.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:21:10.005 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:10.161 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 145 k 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 145 k 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 435 k 2024-03-26T21:21:10.162 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:10.499 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:10.654 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 3.5 M 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T21:21:10.655 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 34 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 16 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-26T21:21:10.656 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:10.657 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 8.6 M 2024-03-26T21:21:10.657 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 104 M 2024-03-26T21:21:10.657 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:10.706 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-19.0.0-2484.g42fc52 266 kB/s | 145 kB 00:00 2024-03-26T21:21:10.707 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:10.707 INFO:teuthology.orchestra.run.smithi067.stdout:Total 266 kB/s | 145 kB 00:00 2024-03-26T21:21:10.707 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:10.715 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:10.715 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:10.769 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:10.770 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:10.924 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:10.965 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:10.990 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:10.990 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T21:21:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:11.201 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2484.g42 168 kB/s | 34 kB 00:00 2024-03-26T21:21:11.317 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2484.g42f 52 kB/s | 16 kB 00:00 2024-03-26T21:21:11.384 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-26T21:21:11.410 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-26T21:21:11.568 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.4 MB/s | 548 kB 00:00 2024-03-26T21:21:11.602 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 583 kB/s | 19 kB 00:00 2024-03-26T21:21:11.710 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-03-26T21:21:11.753 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-03-26T21:21:11.786 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 928 kB/s | 31 kB 00:00 2024-03-26T21:21:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-26T21:21:12.068 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8. 1/1 2024-03-26T21:21:12.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:12.069 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:12.069 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:12.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:12.069 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:12.103 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-03-26T21:21:12.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-03-26T21:21:12.487 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-dashboard-19.0.0-2484.g42fc52 2.4 MB/s | 3.5 MB 00:01 2024-03-26T21:21:12.490 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:12.491 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-03-26T21:21:12.511 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:12.531 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:12.531 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:12.727 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:12.727 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:12.980 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:13.035 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:13.036 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:13.036 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:13.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T21:21:13.265 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:13.347 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T21:21:13.382 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T21:21:13.429 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T21:21:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T21:21:13.558 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T21:21:13.676 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:13.688 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T21:21:13.790 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T21:21:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:13.834 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-26T21:21:13.834 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:13.834 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 3.5 M 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 34 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 16 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T21:21:13.835 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 8.6 M 2024-03-26T21:21:13.837 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 104 M 2024-03-26T21:21:13.837 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:13.838 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T21:21:13.866 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:21:13.899 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 10/12 2024-03-26T21:21:14.260 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2484.g42f 87 kB/s | 16 kB 00:00 2024-03-26T21:21:14.386 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2484.g42 108 kB/s | 34 kB 00:00 2024-03-26T21:21:14.452 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-26T21:21:14.477 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-26T21:21:14.694 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2484.g42fc52b 5.7 MB/s | 3.5 MB 00:00 2024-03-26T21:21:14.903 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 93 kB/s | 19 kB 00:00 2024-03-26T21:21:14.970 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.6 MB/s | 863 kB 00:00 2024-03-26T21:21:14.997 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 11/12 2024-03-26T21:21:15.032 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:21:15.037 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 980 kB/s | 548 kB 00:00 2024-03-26T21:21:15.121 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 528 kB/s | 31 kB 00:00 2024-03-26T21:21:15.238 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.1 MB/s | 2.0 MB 00:00 2024-03-26T21:21:15.305 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-03-26T21:21:15.362 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 402 kB/s | 144 kB 00:00 2024-03-26T21:21:15.362 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:15.363 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.7 MB/s | 8.6 MB 00:01 2024-03-26T21:21:15.363 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:15.372 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:15.372 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:15.572 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T21:21:16.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T21:21:16.280 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T21:21:16.309 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T21:21:16.408 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T21:21:16.492 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:21:16.492 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 1/12 2024-03-26T21:21:16.492 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 2/12 2024-03-26T21:21:16.492 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 3/12 2024-03-26T21:21:16.492 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T21:21:16.493 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T21:21:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T21:21:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T21:21:16.681 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T21:21:16.709 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:21:16.741 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 10/12 2024-03-26T21:21:16.834 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T21:21:16.834 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:21:16.835 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:16.836 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:17.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:21:17.633 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:11 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:17.790 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 7.4 M 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:17.791 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-26T21:21:17.792 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-26T21:21:17.793 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:17.793 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2024-03-26T21:21:17.793 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2024-03-26T21:21:17.793 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 11/12 2024-03-26T21:21:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:21:18.261 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-03-26T21:21:18.303 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2024-03-26T21:21:18.345 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 34 kB 00:00 2024-03-26T21:21:18.395 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T21:21:18.554 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-26T21:21:18.622 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-26T21:21:18.646 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T21:21:18.897 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-03-26T21:21:19.097 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 7.5 MB/s | 7.4 MB 00:00 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 12/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el 1/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noa 2/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9 3/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T21:21:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T21:21:19.314 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 469 kB/s | 195 kB 00:00 2024-03-26T21:21:19.423 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 755 kB/s | 245 kB 00:00 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:19.476 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:19.477 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:19.581 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 861 kB/s | 803 kB 00:00 2024-03-26T21:21:19.583 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:19.584 INFO:teuthology.orchestra.run.smithi055.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-26T21:21:19.628 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:19.654 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:19.654 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:19.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T21:21:20.040 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:20.041 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:20.305 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:10 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:20.463 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 7.4 M 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T21:21:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-03-26T21:21:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:20.466 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 37 M 2024-03-26T21:21:20.466 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2024-03-26T21:21:20.466 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:20.762 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:21:20.845 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 202 kB/s | 32 kB 00:00 2024-03-26T21:21:20.877 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:21:20.887 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 425 kB/s | 17 kB 00:00 2024-03-26T21:21:20.954 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T21:21:20.979 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 377 kB/s | 34 kB 00:00 2024-03-26T21:21:20.984 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:21:21.023 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:21:21.146 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-26T21:21:21.208 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:21:21.213 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-26T21:21:21.238 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:21:21.238 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T21:21:21.317 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:21:21.447 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 67 MB/s | 16 MB 00:00 2024-03-26T21:21:21.647 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 978 kB/s | 195 kB 00:00 2024-03-26T21:21:21.680 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-26T21:21:21.989 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 718 kB/s | 245 kB 00:00 2024-03-26T21:21:22.076 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:21:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 5.3 MB/s | 7.4 MB 00:01 2024-03-26T21:21:22.091 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:22.091 INFO:teuthology.orchestra.run.smithi067.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-26T21:21:22.131 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:22.155 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:22.155 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:22.192 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:21:22.536 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:22.537 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:23.264 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:23.346 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T21:21:23.377 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T21:21:23.474 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:21:23.514 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T21:21:23.569 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:21:23.699 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:21:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:21:23.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T21:21:24.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T21:21:24.619 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T21:21:24.654 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:21:24.687 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:21:25.949 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T21:21:27.037 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T21:21:27.086 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 1/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:21:27.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:21:27.368 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:21:27.798 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:21:27.798 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:27.798 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:27.798 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:27.799 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:28.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:21:28.740 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:22 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:28.898 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:28.899 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:28.899 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 49 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:21:28.900 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:28.901 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2024-03-26T21:21:28.902 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-03-26T21:21:28.902 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:29.222 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 168 kB/s | 24 kB 00:00 2024-03-26T21:21:29.247 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-03-26T21:21:29.305 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T21:21:29.331 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T21:21:29.356 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T21:21:29.381 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-26T21:21:29.431 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): ceph-mgr-rook-19.0.0-2484.g42fc52b8.el9 139 kB/s | 49 kB 00:00 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 12/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc5 1/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T21:21:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T21:21:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T21:21:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T21:21:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T21:21:29.657 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-26T21:21:29.682 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-03-26T21:21:29.891 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-03-26T21:21:29.945 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T21:21:29.945 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:29.946 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:29.958 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 385 kB/s | 203 kB 00:00 2024-03-26T21:21:30.008 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-03-26T21:21:30.283 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-03-26T21:21:30.285 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:30.286 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-26T21:21:30.286 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:30.302 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:30.302 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:30.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-03-26T21:21:30.405 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:30.412 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:30.887 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:20 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:31.043 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 49 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T21:21:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-03-26T21:21:31.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:31.047 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.0 M 2024-03-26T21:21:31.047 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-03-26T21:21:31.047 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:31.060 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:31.287 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:21:31.335 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:21:31.335 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-03-26T21:21:31.352 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-03-26T21:21:31.390 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:21:31.410 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T21:21:31.427 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:21:31.436 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T21:21:31.461 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T21:21:31.468 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:21:31.486 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-26T21:21:31.562 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-26T21:21:31.565 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:21:31.587 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 257 kB/s | 32 kB 00:00 2024-03-26T21:21:31.670 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): ceph-mgr-rook-19.0.0-2484.g42fc52b8.el9 105 kB/s | 49 kB 00:00 2024-03-26T21:21:31.674 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:21:31.696 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.5 MB/s | 203 kB 00:00 2024-03-26T21:21:31.729 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:21:31.738 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-26T21:21:31.771 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-03-26T21:21:31.796 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 470 kB/s | 59 kB 00:00 2024-03-26T21:21:31.798 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:31.799 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-26T21:21:31.799 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:31.813 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:31.813 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:31.911 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:31.911 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:32.430 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:32.484 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:21:32.525 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:21:32.559 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:21:32.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T21:21:32.626 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:21:32.653 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:21:32.665 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T21:21:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T21:21:32.774 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T21:21:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T21:21:32.912 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T21:21:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T21:21:33.093 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T21:21:33.347 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 1/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:21:33.348 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:21:33.697 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:33.698 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:33.854 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T21:21:33.897 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T21:21:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T21:21:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T21:21:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:21:34.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:21:34.628 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:34.761 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 13/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 1/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T21:21:34.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T21:21:34.785 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 141 k 2024-03-26T21:21:34.786 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-26T21:21:34.787 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:34.788 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2024-03-26T21:21:34.788 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2024-03-26T21:21:34.788 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T21:21:35.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:35.126 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:35.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T21:21:35.462 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-26T21:21:35.604 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-26T21:21:35.654 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2484.g42fc52b8.e 442 kB/s | 141 kB 00:00 2024-03-26T21:21:35.940 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:25 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:36.063 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 133 kB/s | 58 kB 00:00 2024-03-26T21:21:36.088 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2024-03-26T21:21:36.096 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 141 k 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T21:21:36.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T21:21:36.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:36.098 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:36.098 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-26T21:21:36.098 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-03-26T21:21:36.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:36.099 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.8 M 2024-03-26T21:21:36.099 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-03-26T21:21:36.099 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:36.205 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 738 kB/s | 548 kB 00:00 2024-03-26T21:21:36.208 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:36.208 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.8 MB/s | 6.8 MB 00:01 2024-03-26T21:21:36.221 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:36.232 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:36.232 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:36.326 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:36.327 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:36.531 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-26T21:21:36.598 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2484.g42fc52b8.e 607 kB/s | 141 kB 00:00 2024-03-26T21:21:36.634 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:36.682 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-26T21:21:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:21:36.882 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:21:36.924 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 179 kB/s | 58 kB 00:00 2024-03-26T21:21:36.974 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-26T21:21:37.099 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 967 kB/s | 548 kB 00:00 2024-03-26T21:21:37.101 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:37.101 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.8 MB/s | 6.8 MB 00:01 2024-03-26T21:21:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:37.216 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:37.216 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:37.436 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:21:37.513 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:37.520 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:21:37.599 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:21:37.627 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:21:37.673 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T21:21:37.727 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T21:21:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T21:21:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T21:21:38.427 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T21:21:38.454 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:21:38.543 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:21:38.543 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/6 2024-03-26T21:21:38.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:21:38.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:21:38.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:21:38.544 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:21:38.846 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:21:38.846 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:38.846 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:38.846 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:38.846 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:38.847 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:39.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 6/6 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 1/6 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T21:21:39.297 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T21:21:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:39.602 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:39.637 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:39.794 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 890 k 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:39.795 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-26T21:21:39.796 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:39.796 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 969 k 2024-03-26T21:21:39.796 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.8 M 2024-03-26T21:21:39.796 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:39.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-03-26T21:21:39.948 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 518 kB/s | 79 kB 00:00 2024-03-26T21:21:40.423 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:40.540 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-19.0.0-2484.g42fc52b8.el9.x86_ 1.2 MB/s | 890 kB 00:00 2024-03-26T21:21:40.541 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:40.542 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.3 MB/s | 969 kB 00:00 2024-03-26T21:21:40.542 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:40.564 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:40.564 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:40.579 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 890 k 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 969 k 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.8 M 2024-03-26T21:21:40.581 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:40.644 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:40.644 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:40.734 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 515 kB/s | 79 kB 00:00 2024-03-26T21:21:40.832 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:40.970 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:21:41.043 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:21:41.360 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-19.0.0-2484.g42fc52b8.el9.x86_ 1.1 MB/s | 890 kB 00:00 2024-03-26T21:21:41.361 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:41.361 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 969 kB 00:00 2024-03-26T21:21:41.361 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:41.384 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:41.384 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:41.464 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:41.464 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:41.659 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:41.790 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T21:21:41.808 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:21:41.808 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/2 2024-03-26T21:21:41.867 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:21:42.079 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:21:42.079 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:42.080 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:42.080 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:21:42.080 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:42.080 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:42.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-volume 2024-03-26T21:21:42.623 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2/2 2024-03-26T21:21:42.624 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/2 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:42.955 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:43.112 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 273 k 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:43.113 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:43.114 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-26T21:21:43.114 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:43.114 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 342 k 2024-03-26T21:21:43.114 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.5 M 2024-03-26T21:21:43.114 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:43.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-03-26T21:21:43.253 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 496 kB/s | 69 kB 00:00 2024-03-26T21:21:43.662 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-volume-19.0.0-2484.g42fc52b8.el9.no 499 kB/s | 273 kB 00:00 2024-03-26T21:21:43.663 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:43.663 INFO:teuthology.orchestra.run.smithi055.stdout:Total 622 kB/s | 342 kB 00:00 2024-03-26T21:21:43.663 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:43.673 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:43.673 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:43.714 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:43.715 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:43.767 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:43.865 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-2484.g42fc52b8.el9 ceph-noarch 273 k 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:43.927 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 342 k 2024-03-26T21:21:43.928 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.5 M 2024-03-26T21:21:43.928 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:44.055 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T21:21:44.063 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 506 kB/s | 69 kB 00:00 2024-03-26T21:21:44.089 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:21:44.104 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:21:44.104 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:44.104 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T21:21:44.104 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:44.172 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-volume-19.0.0-2484.g42fc52b8.el9.no 1.1 MB/s | 273 kB 00:00 2024-03-26T21:21:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.4 MB/s | 342 kB 00:00 2024-03-26T21:21:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:44.220 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:44.220 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:44.576 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T21:21:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:21:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2/2 2024-03-26T21:21:44.623 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:21:44.623 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T21:21:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:44.912 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 1/2 2024-03-26T21:21:45.218 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T21:21:45.218 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:45.218 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:45.218 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:45.219 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T21:21:45.219 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:45.219 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 1/2 2024-03-26T21:21:45.563 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-2484.g42fc52b8.el9.noarch 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:45.658 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:45.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-03-26T21:21:46.143 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:46.301 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 127 k 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:46.302 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 127 k 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 456 k 2024-03-26T21:21:46.303 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:46.573 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:46.729 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 127 k 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 127 k 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 456 k 2024-03-26T21:21:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:46.818 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-19.0.0-2484.g42fc52b8.el9.x86_64 246 kB/s | 127 kB 00:00 2024-03-26T21:21:46.818 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:46.819 INFO:teuthology.orchestra.run.smithi055.stdout:Total 245 kB/s | 127 kB 00:00 2024-03-26T21:21:46.819 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:46.825 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:46.825 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:46.885 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:46.886 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-19.0.0-2484.g42fc52b8.el9.x86_64 546 kB/s | 127 kB 00:00 2024-03-26T21:21:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:Total 542 kB/s | 127 kB 00:00 2024-03-26T21:21:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:46.969 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:46.970 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:47.028 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:47.028 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:47.062 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:47.169 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:47.184 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:47.281 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:47.619 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:47.904 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:48.022 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:48.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2024-03-26T21:21:48.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-03-26T21:21:48.706 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:48.807 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:48.863 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:48.863 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:48.868 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:48.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2024-03-26T21:21:48.968 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:49.025 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:49.026 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:49.026 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:49.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-03-26T21:21:49.495 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:49.650 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:49.651 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:49.651 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-26T21:21:49.651 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:49.651 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:49.651 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 32 k 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 32 k 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 142 k 2024-03-26T21:21:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:49.821 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-19.0.0-2484.g42fc52b8.el9.x86_6 188 kB/s | 32 kB 00:00 2024-03-26T21:21:49.822 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:49.822 INFO:teuthology.orchestra.run.smithi055.stdout:Total 186 kB/s | 32 kB 00:00 2024-03-26T21:21:49.822 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:49.826 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:49.826 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:49.827 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 32 k 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:49.828 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:49.829 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:49.829 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:49.829 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 32 k 2024-03-26T21:21:49.829 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 142 k 2024-03-26T21:21:49.829 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:49.846 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:49.846 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:49.966 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-19.0.0-2484.g42fc52b8.el9.x86_6 187 kB/s | 32 kB 00:00 2024-03-26T21:21:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:50.000 INFO:teuthology.orchestra.run.smithi067.stdout:Total 185 kB/s | 32 kB 00:00 2024-03-26T21:21:50.000 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:50.003 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:50.004 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:50.036 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:50.036 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:50.072 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.466 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.595 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:50.721 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:50.863 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:50.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2024-03-26T21:21:51.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-03-26T21:21:51.456 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:51.555 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:51.602 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:51.610 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:51.611 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:51.611 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:51.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2024-03-26T21:21:51.703 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:51.758 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:51.759 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:51.759 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:51.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-03-26T21:21:52.234 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:52.334 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:52.382 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:52.389 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:52.390 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:52.390 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:52.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2024-03-26T21:21:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:52.537 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:52.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-03-26T21:21:53.016 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:53.115 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:53.169 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:53.169 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:53.170 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:53.171 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:53.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2024-03-26T21:21:53.270 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:53.325 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:53.326 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:53.326 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:53.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-03-26T21:21:53.791 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:53.890 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:53.947 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:53.948 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:53.948 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:53.950 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:53.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2024-03-26T21:21:54.051 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:54.106 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:54.107 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:54.107 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:54.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-03-26T21:21:54.569 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:54.669 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:54.723 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:54.724 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:54.724 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:54.732 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:54.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2024-03-26T21:21:54.832 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:54.886 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:54.887 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:54.888 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:54.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-03-26T21:21:55.354 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:55.462 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:55.509 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:55.517 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:55.518 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-26T21:21:55.518 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:55.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2024-03-26T21:21:55.608 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 is already installed. 2024-03-26T21:21:55.662 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:55.663 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-26T21:21:55.664 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:55.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-03-26T21:21:56.136 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:50 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:56.284 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:56.291 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:56.291 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 86 k 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:56.293 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 86 k 2024-03-26T21:21:56.293 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 231 k 2024-03-26T21:21:56.293 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:56.440 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:56.440 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:56.440 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 86 k 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 86 k 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 231 k 2024-03-26T21:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:57.135 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 124 kB/s | 86 kB 00:00 2024-03-26T21:21:57.136 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:57.136 INFO:teuthology.orchestra.run.smithi067.stdout:Total 123 kB/s | 86 kB 00:00 2024-03-26T21:21:57.136 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:57.144 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:57.144 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:21:57.203 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:21:57.204 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:21:57.233 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 91 kB/s | 86 kB 00:00 2024-03-26T21:21:57.234 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:57.234 INFO:teuthology.orchestra.run.smithi055.stdout:Total 91 kB/s | 86 kB 00:00 2024-03-26T21:21:57.235 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:21:57.244 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:21:57.244 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:21:57.306 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:21:57.306 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:21:57.354 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:21:57.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:57.464 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:21:57.560 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:57.838 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:57.974 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:58.147 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:21:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:21:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:58.283 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:21:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:21:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:58.284 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:21:58.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-03-26T21:21:58.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2024-03-26T21:21:58.952 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:21:59.055 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:53 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:21:59.107 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:21:59.108 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:21:59.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:21:59.109 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.1 M 2024-03-26T21:21:59.109 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 13 M 2024-03-26T21:21:59.109 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 3.1 M 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:59.210 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.1 M 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 13 M 2024-03-26T21:21:59.211 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:21:59.938 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 3.7 MB/s | 3.1 MB 00:00 2024-03-26T21:21:59.938 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:21:59.939 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.7 MB/s | 3.1 MB 00:00 2024-03-26T21:21:59.939 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:21:59.949 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:21:59.949 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:22:00.026 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:22:00.027 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:22:00.044 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 3.7 MB/s | 3.1 MB 00:00 2024-03-26T21:22:00.045 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:22:00.045 INFO:teuthology.orchestra.run.smithi055.stdout:Total 3.7 MB/s | 3.1 MB 00:00 2024-03-26T21:22:00.045 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:22:00.057 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:22:00.058 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:22:00.137 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:22:00.137 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:22:00.463 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:22:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:22:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:00.583 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:22:00.653 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:00.677 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:00.677 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T21:22:00.678 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T21:22:00.678 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:22:00.678 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T21:22:00.678 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:01.639 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:22:01.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-03-26T21:22:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:01.891 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:01.892 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:22:01.892 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:22:01.892 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:01.892 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:22:02.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2024-03-26T21:22:02.454 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 09:20:10 PM UTC. 2024-03-26T21:22:02.609 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 171 k 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 171 k 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 494 k 2024-03-26T21:22:02.610 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-26T21:22:02.732 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:56 ago on Tue 26 Mar 2024 09:20:06 PM UTC. 2024-03-26T21:22:02.886 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:19.0.0-2484.g42fc52b8.el9 ceph 171 k 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-26T21:22:02.887 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-26T21:22:02.888 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:02.888 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 171 k 2024-03-26T21:22:02.889 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 494 k 2024-03-26T21:22:02.889 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-26T21:22:03.481 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 288 kB/s | 171 kB 00:00 2024-03-26T21:22:03.482 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:22:03.482 INFO:teuthology.orchestra.run.smithi055.stdout:Total 287 kB/s | 171 kB 00:00 2024-03-26T21:22:03.482 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-26T21:22:03.491 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-26T21:22:03.491 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-26T21:22:03.550 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-26T21:22:03.551 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-26T21:22:03.701 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-26T21:22:03.807 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:04.236 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:04.382 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-19.0.0-2484.g42fc52b8.el9.x86_64.rpm 96 kB/s | 171 kB 00:01 2024-03-26T21:22:04.382 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:22:04.383 INFO:teuthology.orchestra.run.smithi067.stdout:Total 96 kB/s | 171 kB 00:01 2024-03-26T21:22:04.383 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-26T21:22:04.394 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-26T21:22:04.394 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-26T21:22:04.457 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-26T21:22:04.457 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-26T21:22:04.609 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-26T21:22:04.700 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:04.734 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:05.096 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 1/1 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-2484.g42fc52b8.el9.x86_64 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:22:05.378 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-26T21:22:05.655 DEBUG:teuthology.parallel:result is None 2024-03-26T21:22:05.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:06.042 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:22:06.066 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-2484.g42fc52b8.el9 2024-03-26T21:22:06.067 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2484.g42fc52b8.el9 2024-03-26T21:22:06.067 INFO:teuthology.task.install:The correct ceph version 19.0.0-2484.g42fc52b8 is installed. 2024-03-26T21:22:06.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:06.210 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T21:22:06.232 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-2484.g42fc52b8.el9 2024-03-26T21:22:06.232 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2484.g42fc52b8.el9 2024-03-26T21:22:06.232 INFO:teuthology.task.install:The correct ceph version 19.0.0-2484.g42fc52b8 is installed. 2024-03-26T21:22:06.234 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T21:22:06.234 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:22:06.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:22:06.266 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:22:06.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T21:22:06.300 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T21:22:06.300 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:22:06.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:22:06.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:22:06.404 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:22:06.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T21:22:06.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T21:22:06.499 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T21:22:06.499 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:22:06.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:22:06.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:22:06.595 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:22:06.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T21:22:06.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T21:22:06.688 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T21:22:06.688 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:22:06.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:22:06.715 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:22:06.782 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:22:06.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T21:22:06.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T21:22:06.875 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T21:22:06.968 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'cephadm_mode': 'root'} 2024-03-26T21:22:06.968 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:06.969 INFO:tasks.cephadm:Cluster fsid is e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:06.969 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T21:22:06.969 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.55', 'mon.c': '[v2:172.21.15.55:3301,v1:172.21.15.55:6790]', 'mon.b': '172.21.15.67'} 2024-03-26T21:22:06.969 INFO:tasks.cephadm:First mon is mon.a on smithi055 2024-03-26T21:22:06.969 INFO:tasks.cephadm:First mgr is y 2024-03-26T21:22:06.969 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T21:22:06.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2024-03-26T21:22:06.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-03-26T21:22:07.024 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T21:22:07.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:07.400 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77974/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2484-g42fc52b8', 'node_name': '172.21.4.61+sulcata01', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2484.g42fc52b8'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-26 18:30:27.407480', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-2024-03-26-0925', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-03-26T21:22:07.788 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-2024-03-26-0925, sha1 42fc52b86b14d4f88c40d75eb7102e0ab3918f55 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:22:07.789 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/x86_64/flavors/default/cephadm 2024-03-26T21:22:07.789 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:11.526 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-r--r--. 1 ubuntu ubuntu 781453 Mar 26 21:22 /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:11.526 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-2024-03-26-0925/42fc52b86b14d4f88c40d75eb7102e0ab3918f55/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:12.546 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 781453 Mar 26 21:22 /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:12.546 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:12.565 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T21:22:12.596 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 on all hosts... 2024-03-26T21:22:12.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 pull 2024-03-26T21:22:12.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 pull 2024-03-26T21:22:13.508 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:22:13.600 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:22:46.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:22:46.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev)", 2024-03-26T21:22:46.701 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063", 2024-03-26T21:22:46.701 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-03-26T21:22:46.701 INFO:teuthology.orchestra.run.smithi067.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d" 2024-03-26T21:22:46.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:22:46.701 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev)", 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063", 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d" 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:22:47.773 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:22:47.798 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2024-03-26T21:22:47.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-03-26T21:22:47.885 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2024-03-26T21:22:47.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-03-26T21:22:47.956 INFO:tasks.cephadm:Writing seed config... 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T21:22:47.957 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-26T21:22:47.958 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T21:22:47.959 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-26T21:22:47.959 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:22:47.959 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T21:22:47.974 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e5f52b02-ebb6-11ee-95d0-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-26T21:22:47.974 DEBUG:teuthology.orchestra.run.smithi055:mon.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T21:22:48.016 DEBUG:teuthology.orchestra.run.smithi055:mgr.y> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T21:22:48.059 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T21:22:48.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 -v bootstrap --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.55 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:22:48.256 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-26T21:22:48.256 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55', '-v', 'bootstrap', '--fsid', 'e5f52b02-ebb6-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.55', '--skip-admin-label'] 2024-03-26T21:22:48.257 INFO:teuthology.orchestra.run.smithi055.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T21:22:48.257 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying podman|docker is present... 2024-03-26T21:22:48.281 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:22:48.281 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying lvm2 is present... 2024-03-26T21:22:48.281 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying time synchronization is in place... 2024-03-26T21:22:48.288 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:22:48.288 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:22:48.294 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:22:48.294 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:22:48.300 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2024-03-26T21:22:48.306 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2024-03-26T21:22:48.306 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:22:48.306 INFO:teuthology.orchestra.run.smithi055.stdout:Repeating the final host check... 2024-03-26T21:22:48.330 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T21:22:48.330 INFO:teuthology.orchestra.run.smithi055.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T21:22:48.330 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl is present 2024-03-26T21:22:48.330 INFO:teuthology.orchestra.run.smithi055.stdout:lvcreate is present 2024-03-26T21:22:48.336 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T21:22:48.336 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T21:22:48.342 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T21:22:48.342 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:22:48.348 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:Host looks OK 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:Cluster fsid: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:Acquiring lock 139720830915248 on /run/cephadm/e5f52b02-ebb6-11ee-95d0-87774f69a715.lock 2024-03-26T21:22:48.354 INFO:teuthology.orchestra.run.smithi055.stdout:Lock 139720830915248 acquired on /run/cephadm/e5f52b02-ebb6-11ee-95d0-87774f69a715.lock 2024-03-26T21:22:48.355 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 3300 ... 2024-03-26T21:22:48.355 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 6789 ... 2024-03-26T21:22:48.355 INFO:teuthology.orchestra.run.smithi055.stdout:Base mon IP(s) is [172.21.15.55:3300, 172.21.15.55:6789], mon addrv is [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2024-03-26T21:22:48.358 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.55 metric 100 2024-03-26T21:22:48.358 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.55 metric 100 2024-03-26T21:22:48.360 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T21:22:48.361 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T21:22:48.361 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2024-03-26T21:22:48.363 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T21:22:48.364 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2024-03-26T21:22:48.364 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2024-03-26T21:22:48.364 INFO:teuthology.orchestra.run.smithi055.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T21:22:48.364 INFO:teuthology.orchestra.run.smithi055.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T21:22:48.365 INFO:teuthology.orchestra.run.smithi055.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55... 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:2e177a2f3361febef0925bf17c6fa96e6cd4af765a1dda3326a2b328e0c37fe9 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying config sha256:a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 2024-03-26T21:22:48.759 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T21:22:49.613 INFO:teuthology.orchestra.run.smithi055.stdout:ceph: stdout ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev) 2024-03-26T21:22:49.614 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph version: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev) 2024-03-26T21:22:49.614 INFO:teuthology.orchestra.run.smithi055.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T21:22:50.538 INFO:teuthology.orchestra.run.smithi055.stdout:stat: stdout 167 167 2024-03-26T21:22:50.538 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial keys... 2024-03-26T21:22:51.495 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQCqPANmHl2aMxAACkx1Cq+YwNive8G7Aj/zgg== 2024-03-26T21:22:52.452 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQCrPANmXb5sMRAAq4aLMJYbv0KfyjIOagBjpQ== 2024-03-26T21:22:53.292 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQCsPANmhfVULhAAWKRUfuhP69jBHBHLeL1dPg== 2024-03-26T21:22:53.293 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial monmap... 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool for a [v2:172.21.15.55:3300,v1:172.21.15.55:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = quincy 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: set fsid to e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T21:22:54.200 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:22:54.201 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mon... 2024-03-26T21:22:55.098 INFO:teuthology.orchestra.run.smithi055.stdout:create mon.a on 2024-03-26T21:22:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T21:22:55.474 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T21:22:55.650 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target. 2024-03-26T21:22:55.650 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target. 2024-03-26T21:22:55.932 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a 2024-03-26T21:22:55.932 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T21:22:56.093 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target.wants/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@.service. 2024-03-26T21:22:56.381 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 podman[29018]: 2024-03-26 21:22:56.387946769 +0000 UTC m=+0.166824518 container create d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:22:56.642 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 podman[29018]: 2024-03-26 21:22:56.582530296 +0000 UTC m=+0.361408061 container init d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:22:56.642 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 podman[29018]: 2024-03-26 21:22:56.585800329 +0000 UTC m=+0.364678085 container start d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:22:56.655 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:22:56.655 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-03-26T21:22:56.662 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:22:56.662 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:22:56.662 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-03-26T21:22:56.662 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:22:56.663 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon to start... 2024-03-26T21:22:56.663 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon... 2024-03-26T21:22:56.898 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 bash[29018]: d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d 2024-03-26T21:22:56.898 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 systemd[1]: Started Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:22:56.898 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 ceph-mon[29038]: mkfs e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:56.898 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:56 smithi055 ceph-mon[29038]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:22:57.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T21:22:57.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout id: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout services: 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.507195s) 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout data: 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T21:22:57.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T21:22:57.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:57.689 INFO:teuthology.orchestra.run.smithi055.stdout:mon is available 2024-03-26T21:22:57.689 INFO:teuthology.orchestra.run.smithi055.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T21:22:57.808 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: monmap epoch 1 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: last_changed 2024-03-26T21:22:53.649249+0000 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: min_mon_release 19 (squid) 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: election_strategy: 1 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: fsmap 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: mgrmap e1: no daemons active 2024-03-26T21:22:57.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:57 smithi055 ceph-mon[29038]: from='client.? 172.21.15.55:0/501564148' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:22:58.761 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:22:58.762 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:22:58.762 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:22:58.762 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:22:58.762 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:22:58.762 INFO:teuthology.orchestra.run.smithi055.stdout:Generating new minimal ceph.conf... 2024-03-26T21:22:58.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:58 smithi055 ceph-mon[29038]: from='client.? 172.21.15.55:0/3023587748' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:22:58.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:58 smithi055 ceph-mon[29038]: from='client.? 172.21.15.55:0/3023587748' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T21:22:59.767 INFO:teuthology.orchestra.run.smithi055.stdout:Restarting the monitor... 2024-03-26T21:22:59.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:59 smithi055 ceph-mon[29038]: from='client.? 172.21.15.55:0/3287449401' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:23:00.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:22:59 smithi055 systemd[1]: Stopping Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:23:00.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29033]: 2024-03-26T21:23:00.032+0000 7f98ceb2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:23:00.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29033]: 2024-03-26T21:23:00.032+0000 7f98ceb2f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T21:23:00.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 podman[29282]: 2024-03-26 21:23:00.217211987 +0000 UTC m=+0.272990363 container died d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-26T21:23:00.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 podman[29282]: 2024-03-26 21:23:00.54937073 +0000 UTC m=+0.605149096 container cleanup d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:23:00.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 bash[29282]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a 2024-03-26T21:23:00.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 podman[29294]: 2024-03-26 21:23:00.725637622 +0000 UTC m=+0.509751642 container remove d479f4964e1763d4c625f51143aca10597876e1fcb4959e5e2ac9e8c0284a02d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:23:01.272 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T21:23:01.272 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 systemd[1]: Stopped Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:23:01.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:00 smithi055 systemd[1]: Starting Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:23:01.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 podman[29386]: 2024-03-26 21:23:01.176458347 +0000 UTC m=+0.112331319 container create a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-03-26T21:23:01.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 podman[29386]: 2024-03-26 21:23:01.081653184 +0000 UTC m=+0.017526167 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 2024-03-26T21:23:01.450 INFO:teuthology.orchestra.run.smithi055.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 podman[29386]: 2024-03-26 21:23:01.370695848 +0000 UTC m=+0.306568821 container init a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 podman[29386]: 2024-03-26 21:23:01.373486273 +0000 UTC m=+0.309359251 container start a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev), process ceph-mon, pid 2 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: pidfile_write: ignore empty --pid-file 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: load: jerasure load: lrc 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Git sha 0 2024-03-26T21:23:01.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Compile date 2024-03-26 11:42:14 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: DB SUMMARY 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: DB Session ID: Y67LS9TG4ZH6CLR8WVXB 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 96177 ; 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:23:01.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.env: 0x55f2404d2c60 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.info_log: 0x55f2429f1ae0 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.statistics: (nil) 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.use_fsync: 0 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:23:01.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.db_log_dir: 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.wal_dir: 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:23:01.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.write_buffer_manager: 0x55f2429c21e0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.unordered_write: 0 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:23:01.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.row_cache: None 2024-03-26T21:23:01.545 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.wal_filter: None 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.wal_compression: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:23:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:23:01.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_open_files: -1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Compression algorithms supported: 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kZSTD supported: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:23:01.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.merge_operator: 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_filter: None 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f2429f1cc0) 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: index_type: 0 2024-03-26T21:23:01.549 INFO:journalctl@ceph.mon.a.smithi055.stdout: data_block_index_type: 0 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: index_shortening: 1 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: checksum: 4 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: no_block_cache: 0 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_cache: 0x55f2429f5090 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_cache_options: 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: capacity : 536870912 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: num_shard_bits : 4 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: strict_capacity_limit : 0 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_cache_compressed: (nil) 2024-03-26T21:23:01.550 INFO:journalctl@ceph.mon.a.smithi055.stdout: persistent_cache: (nil) 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_size: 4096 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_size_deviation: 10 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_restart_interval: 16 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: index_block_restart_interval: 1 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: metadata_block_size: 4096 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: partition_filters: 0 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: use_delta_encoding: 1 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: filter_policy: bloomfilter 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: whole_key_filtering: 1 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: verify_compression: 0 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: format_version: 5 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: enable_index_compression: 1 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: block_align: 0 2024-03-26T21:23:01.551 INFO:journalctl@ceph.mon.a.smithi055.stdout: max_auto_readahead_size: 262144 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout: prepopulate_block_cache: 0 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression: NoCompression 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.num_levels: 7 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:23:01.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:23:01.553 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:23:01.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:23:01.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:23:01.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.ttl: 2592000 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:23:01.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7a46c402-0ae3-4988-8f3b-7b1b4aceabf1 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488181403212, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488181404184, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92042, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 288, "table_properties": {"data_size": 90090, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11976, "raw_average_key_size": 48, "raw_value_size": 83523, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 247, "num_filter_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711488181, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7a46c402-0ae3-4988-8f3b-7b1b4aceabf1", "db_session_id": "Y67LS9TG4ZH6CLR8WVXB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488181404274, "job": 1, "event": "recovery_finished"} 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:23:01.558 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f242ae4000 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: DB pointer 0x55f242ace000 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: ** DB Stats ** 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:23:01.559 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: L0 2/0 91.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Sum 2/0 91.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:23:01.560 INFO:journalctl@ceph.mon.a.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: Cumulative compaction: 0.00 GB write, 23.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: Interval compaction: 0.00 GB write, 23.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: Block cache BinnedLRUCache@0x55f2429f5090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: starting mon.a rank 0 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???) e1 preinit fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).mds e1 new map 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).mds e1 print_map 2024-03-26T21:23:01.561 INFO:journalctl@ceph.mon.a.smithi055.stdout: e1 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout: legacy client fscid: -1 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout: 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout: No filesystems configured 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 bash[29386]: a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 systemd[1]: Started Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: monmap epoch 1 2024-03-26T21:23:01.562 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: last_changed 2024-03-26T21:22:53.649249+0000 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: min_mon_release 19 (squid) 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: election_strategy: 1 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: fsmap 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T21:23:01.563 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:01 smithi055 ceph-mon[29406]: mgrmap e1: no daemons active 2024-03-26T21:23:02.491 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T21:23:02.512 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:23:02.513 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mgr... 2024-03-26T21:23:02.513 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T21:23:02.513 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T21:23:02.765 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y 2024-03-26T21:23:02.765 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T21:23:02.913 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715.target.wants/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@.service. 2024-03-26T21:23:03.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:02 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/816963101' entity='client.admin' 2024-03-26T21:23:03.431 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:23:03.431 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-03-26T21:23:03.438 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:23:03.438 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:23:03.438 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-03-26T21:23:03.439 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T21:23:03.446 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:23:03.446 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr to start... 2024-03-26T21:23:03.454 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr... 2024-03-26T21:23:04.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:03 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/907708913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:23:04.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:04 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:04.030+0000 7f874614f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:23:04.506 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:04.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:23:04.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:23:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:23:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:22:56.674170+0000", 2024-03-26T21:23:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:23:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (1/15)... 2024-03-26T21:23:04.786 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:04 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:04.699+0000 7f874614f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:23:04.787 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:04 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:04.794+0000 7f874614f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:23:05.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:04 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:04.936+0000 7f874614f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:23:05.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.034+0000 7f874614f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:23:05.388 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.119+0000 7f874614f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:23:05.389 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.396+0000 7f874614f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:23:05.685 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.509+0000 7f874614f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:05.685 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.596+0000 7f874614f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:23:05.685 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.692+0000 7f874614f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:23:06.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:05.870+0000 7f874614f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:23:07.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2043001120' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:23:07.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:06 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:06.878+0000 7f874614f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:23:07.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:07.014+0000 7f874614f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:23:07.547 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:07.547 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:07.547 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:23:07.547 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:23:07.547 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:23:07.548 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:23:07.549 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:23:07.553 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:23:07.554 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:23:07.555 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:22:56.674170+0000", 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:07.556 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (2/15)... 2024-03-26T21:23:07.621 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:07.360+0000 7f874614f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:23:07.621 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:07.451+0000 7f874614f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:23:08.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:07.652+0000 7f874614f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:08.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:07.841+0000 7f874614f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:23:08.408 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:08.132+0000 7f874614f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:08.408 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:08.415+0000 7f874614f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:08.815 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:08.658+0000 7f874614f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:23:09.775 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:09.486+0000 7f874614f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:23:10.393 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3807168874' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:23:10.393 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:10.073+0000 7f874614f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:23:10.393 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:10.180+0000 7f874614f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:23:10.393 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:10.400+0000 7f874614f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: Activating manager daemon y 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: mgrmap e2: y(active, starting, since 0.00330324s) 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:23:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: Manager daemon y is now available 2024-03-26T21:23:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:23:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:23:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' 2024-03-26T21:23:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' 2024-03-26T21:23:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:11 smithi055 ceph-mon[29406]: from='mgr.14100 172.21.15.55:0/364350875' entity='mgr.y' 2024-03-26T21:23:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:12 smithi055 ceph-mon[29406]: mgrmap e3: y(active, since 1.01173s) 2024-03-26T21:23:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:23:13.210 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:23:13.211 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:23:13.212 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:23:13.215 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:13.216 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:23:13.217 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:22:56.674170+0000", 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:13.218 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (3/15)... 2024-03-26T21:23:14.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:14 smithi055 ceph-mon[29406]: mgrmap e4: y(active, since 3s) 2024-03-26T21:23:19.509 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4114516947' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T21:23:21.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:21.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:21.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-03-26T21:23:21.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T21:23:21.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T21:23:21.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T21:23:21.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T21:22:56.674170+0000", 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:21.844 INFO:teuthology.orchestra.run.smithi055.stdout:mgr is available 2024-03-26T21:23:22.762 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3875913806' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T21:23:22.992 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T21:23:22.993 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2024-03-26T21:23:22.993 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T21:23:22.993 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T21:23:22.993 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T21:23:22.993 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling cephadm module... 2024-03-26T21:23:24.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2404993077' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T21:23:24.012 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:23 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setuser ceph since I am not root 2024-03-26T21:23:24.012 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:23 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setgroup ceph since I am not root 2024-03-26T21:23:24.517 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:24 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:24.272+0000 7f3bd8e48200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:23:24.517 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:24 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:24.377+0000 7f3bd8e48200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:23:24.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:24 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2404993077' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T21:23:24.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:24 smithi055 ceph-mon[29406]: mgrmap e5: y(active, since 13s) 2024-03-26T21:23:25.285 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.053+0000 7f3bd8e48200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:23:25.285 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.148+0000 7f3bd8e48200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:23:25.285 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.291+0000 7f3bd8e48200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:23:25.565 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.389+0000 7f3bd8e48200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:23:25.565 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.473+0000 7f3bd8e48200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:23:26.044 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1687744471' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:23:26.044 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.750+0000 7f3bd8e48200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:23:26.045 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.865+0000 7f3bd8e48200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:26.045 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:25.953+0000 7f3bd8e48200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:23:26.045 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:26 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:26.050+0000 7f3bd8e48200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2024-03-26T21:23:26.124 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 5... 2024-03-26T21:23:26.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:26 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:26.238+0000 7f3bd8e48200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:23:27.566 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:27 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:27.225+0000 7f3bd8e48200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:23:27.566 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:27 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:27.359+0000 7f3bd8e48200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:27 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:27.696+0000 7f3bd8e48200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:27 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:27.789+0000 7f3bd8e48200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:23:27.984 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:27 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:27.990+0000 7f3bd8e48200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:28.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:28.181+0000 7f3bd8e48200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:23:28.730 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:28.475+0000 7f3bd8e48200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:28.730 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:28.736+0000 7f3bd8e48200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:29.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:28.982+0000 7f3bd8e48200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:23:30.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:29 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:29.815+0000 7f3bd8e48200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:23:30.719 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:30.400+0000 7f3bd8e48200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:23:30.719 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:30.502+0000 7f3bd8e48200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:23:30.719 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:30.726+0000 7f3bd8e48200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: Active manager daemon y restarted 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: Activating manager daemon y 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: mgrmap e6: y(active, starting, since 0.00358572s) 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:23:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: Manager daemon y is now available 2024-03-26T21:23:31.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:31.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:31.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:31.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:30 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:32.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:31 smithi055 ceph-mon[29406]: Found migration_current of "None". Setting to last migration. 2024-03-26T21:23:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:31 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:23:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:31 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:23:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:31 smithi055 ceph-mon[29406]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:23:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:31 smithi055 ceph-mon[29406]: mgrmap e7: y(active, since 1.00716s) 2024-03-26T21:23:32.256 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:32.256 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T21:23:32.256 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:23:32.257 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:32.257 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 5 is available 2024-03-26T21:23:32.257 INFO:teuthology.orchestra.run.smithi055.stdout:Setting orchestrator backend to cephadm... 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:32] ENGINE Bus STARTING 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:33.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:33 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:34 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:32] ENGINE Serving on https://172.21.15.55:7150 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:34 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:32] ENGINE Serving on http://172.21.15.55:8765 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:34 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:32] ENGINE Bus STARTED 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:34 smithi055 ceph-mon[29406]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:34.426 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:34 smithi055 ceph-mon[29406]: mgrmap e8: y(active, since 2s) 2024-03-26T21:23:34.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T21:23:34.688 INFO:teuthology.orchestra.run.smithi055.stdout:Generating ssh key... 2024-03-26T21:23:35.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-mon[29406]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: Generating public/private rsa key pair. 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: Your identification has been saved in /tmp/tmpmyhjdqr6/key. 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: Your public key has been saved in /tmp/tmpmyhjdqr6/key.pub. 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: The key fingerprint is: 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: SHA256:MBnUXjwdErfxw+IXhiWeuPxkzDQNX90JraXY6iQdDd8 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:35.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: The key's randomart image is: 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: +---[RSA 3072]----+ 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | .o. .oo*o+ =| 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | o. +*.& =o| 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | +. ...# & | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | o.. O X E | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | S + O . | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | . B . | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | + . | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | . | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: | | 2024-03-26T21:23:35.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: +----[SHA256]-----+ 2024-03-26T21:23:36.767 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:36 smithi055 ceph-mon[29406]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:36.767 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:36 smithi055 ceph-mon[29406]: Generating ssh key... 2024-03-26T21:23:36.767 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:36 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:36.767 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:36 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:37.235 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpITC1mxHZ5j832LwvPCR1yWJlT+sGQ4zxvUoWarZU8AzXtUi4nvREiVv9Q52Wc+bZJYhc2FfVr+OdotcdnXOF59rlAe4mH3EOUHcMcNWagP4QaVYnYqxxLWF9+OdxXdS9Ko4y7pV9ndkRXkYBwXUju9yWf/hXcCuDlR97Z9j6TWLPDBRCKJbxHbUOzVlWDiEqoPwt9u68zzkWHdRSr57ClsW7hHBspRnre2ve90Q7yPHE9HdrJgqbxgPbFt88LcwT8L7oafy6KwiWR6f9iIYf9pRD9uN6YWf7581x5mJvkZVpLXHPf55suNmzjEjOXPsnFLz57rAEvOzm+/XpiTkq28y0BPFCocYtEz5xSyDA98G2ThanNChA/vdES+evTmq+niVLFAUz74IC640e1H767yaZ14lWcfz7Vm1yeAHGNARHFq3IcNOy3mZlY1+4dHJW0EAnMfwwPzkthnZtnFbtFucKTtA/SQMKjk6Bppbg5qXBKzU8njsw+QZha87aHks= ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:23:37.236 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:23:37.236 INFO:teuthology.orchestra.run.smithi055.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T21:23:37.236 INFO:teuthology.orchestra.run.smithi055.stdout:Adding host smithi055... 2024-03-26T21:23:38.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:37 smithi055 ceph-mon[29406]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:39.022 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:38 smithi055 ceph-mon[29406]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:39.023 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:38 smithi055 ceph-mon[29406]: Deploying cephadm binary to smithi055 2024-03-26T21:23:42.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Added host 'smithi055' with addr '172.21.15.55' 2024-03-26T21:23:42.838 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying unmanaged mon service... 2024-03-26T21:23:43.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:43 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:43.475 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:43 smithi055 ceph-mon[29406]: Added host smithi055 2024-03-26T21:23:43.475 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:43 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:44.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:44 smithi055 ceph-mon[29406]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:44.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:44 smithi055 ceph-mon[29406]: Saving service mon spec with placement count:5 2024-03-26T21:23:44.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:44 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:45.046 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T21:23:45.047 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying unmanaged mgr service... 2024-03-26T21:23:46.025 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:45 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:46.025 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:45 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:46.356 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T21:23:47.029 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:46 smithi055 ceph-mon[29406]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:23:47.030 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:46 smithi055 ceph-mon[29406]: Saving service mgr spec with placement count:2 2024-03-26T21:23:47.030 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:46 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:48.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3275160077' entity='client.admin' 2024-03-26T21:23:48.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:48 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:49.264 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling the dashboard module... 2024-03-26T21:23:49.968 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/488810238' entity='client.admin' 2024-03-26T21:23:49.968 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:49 smithi055 ceph-mon[29406]: from='mgr.14120 172.21.15.55:0/147534491' entity='mgr.y' 2024-03-26T21:23:51.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2489970480' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T21:23:51.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:50 smithi055 ceph-mon[29406]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:23:51.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:50 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setuser ceph since I am not root 2024-03-26T21:23:51.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:50 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setgroup ceph since I am not root 2024-03-26T21:23:51.610 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:51 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:51.359+0000 7f72c9b41200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:23:51.611 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:51 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:51.460+0000 7f72c9b41200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:23:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2489970480' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T21:23:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:51 smithi055 ceph-mon[29406]: mgrmap e9: y(active, since 20s) 2024-03-26T21:23:52.368 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.133+0000 7f72c9b41200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:23:52.368 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.229+0000 7f72c9b41200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:23:52.368 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.375+0000 7f72c9b41200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:23:52.672 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.473+0000 7f72c9b41200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:23:52.672 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.558+0000 7f72c9b41200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:23:52.675 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2024-03-26T21:23:52.676 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 9... 2024-03-26T21:23:52.938 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3622590580' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T21:23:52.938 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.831+0000 7f72c9b41200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:23:52.938 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:52 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:52.945+0000 7f72c9b41200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:53.305 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:53 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:53.035+0000 7f72c9b41200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:23:53.306 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:53 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:53.134+0000 7f72c9b41200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:23:53.306 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:53 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:53.313+0000 7f72c9b41200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:23:54.566 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:54 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:54.292+0000 7f72c9b41200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:23:54.566 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:54 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:54.425+0000 7f72c9b41200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:23:55.049 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:54 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:54.763+0000 7f72c9b41200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:23:55.049 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:54 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:54.855+0000 7f72c9b41200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:23:55.049 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:55 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:55.056+0000 7f72c9b41200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:55.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:55 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:55.245+0000 7f72c9b41200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:23:55.790 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:55 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:55.537+0000 7f72c9b41200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:23:55.790 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:55 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:55.798+0000 7f72c9b41200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:23:56.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:56 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:56.042+0000 7f72c9b41200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:23:57.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:56 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:56.867+0000 7f72c9b41200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:23:57.766 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:57.449+0000 7f72c9b41200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:23:57.767 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:57.549+0000 7f72c9b41200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:23:57.767 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:23:57.774+0000 7f72c9b41200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: Active manager daemon y restarted 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: Activating manager daemon y 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: mgrmap e10: y(active, starting, since 0.00384753s) 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: Manager daemon y is now available 2024-03-26T21:23:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:57 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:58] ENGINE Bus STARTING 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: mgrmap e11: y(active, since 1.00774s) 2024-03-26T21:23:59.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:58 smithi055 ceph-mon[29406]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 9 is available 2024-03-26T21:23:59.283 INFO:teuthology.orchestra.run.smithi055.stdout:Generating a dashboard self-signed certificate... 2024-03-26T21:24:00.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:59 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:58] ENGINE Serving on http://172.21.15.55:8765 2024-03-26T21:24:00.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:59 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:59] ENGINE Serving on https://172.21.15.55:7150 2024-03-26T21:24:00.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:23:59 smithi055 ceph-mon[29406]: [26/Mar/2024:21:23:59] ENGINE Bus STARTED 2024-03-26T21:24:00.566 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T21:24:00.566 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial admin user... 2024-03-26T21:24:00.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:00 smithi055 ceph-mon[29406]: mgrmap e12: y(active, since 2s) 2024-03-26T21:24:00.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:00 smithi055 ceph-mon[29406]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:00.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:00.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:00.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:02.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Fr9O3PVRC/DPhSlYD.BFcOxWBbagUMZFY2UY7OGBA3C8pfEjf0JRm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711488241, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T21:24:02.031 INFO:teuthology.orchestra.run.smithi055.stdout:Fetching dashboard port number... 2024-03-26T21:24:02.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:02 smithi055 ceph-mon[29406]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:03 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/809020153' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T21:24:03.658 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T21:24:03.668 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T21:24:03.668 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2024-03-26T21:24:03.676 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T21:24:03.676 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2024-03-26T21:24:03.676 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2024-03-26T21:24:03.676 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T21:24:03.677 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph Dashboard is now available at: 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout: URL: https://smithi055.front.sepia.ceph.com:8443/ 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout: User: admin 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout: Password: qcf9s67w72 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:03.678 INFO:teuthology.orchestra.run.smithi055.stdout:Saving cluster configuration to /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config directory 2024-03-26T21:24:03.679 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling autotune for osd_memory_target 2024-03-26T21:24:05.374 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:05 smithi055 ceph-mon[29406]: mgrmap e13: y(active, since 6s) 2024-03-26T21:24:06.457 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T21:24:06.457 INFO:teuthology.orchestra.run.smithi055.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: ceph telemetry on 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout:For more information see: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.458 INFO:teuthology.orchestra.run.smithi055.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T21:24:06.459 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:06.459 INFO:teuthology.orchestra.run.smithi055.stdout:Bootstrap complete. 2024-03-26T21:24:06.504 INFO:tasks.cephadm:Fetching config... 2024-03-26T21:24:06.504 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:06.504 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T21:24:06.520 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T21:24:06.520 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:06.520 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T21:24:06.575 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T21:24:06.575 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:06.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T21:24:06.642 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T21:24:06.642 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:06.642 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T21:24:06.697 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T21:24:06.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpITC1mxHZ5j832LwvPCR1yWJlT+sGQ4zxvUoWarZU8AzXtUi4nvREiVv9Q52Wc+bZJYhc2FfVr+OdotcdnXOF59rlAe4mH3EOUHcMcNWagP4QaVYnYqxxLWF9+OdxXdS9Ko4y7pV9ndkRXkYBwXUju9yWf/hXcCuDlR97Z9j6TWLPDBRCKJbxHbUOzVlWDiEqoPwt9u68zzkWHdRSr57ClsW7hHBspRnre2ve90Q7yPHE9HdrJgqbxgPbFt88LcwT8L7oafy6KwiWR6f9iIYf9pRD9uN6YWf7581x5mJvkZVpLXHPf55suNmzjEjOXPsnFLz57rAEvOzm+/XpiTkq28y0BPFCocYtEz5xSyDA98G2ThanNChA/vdES+evTmq+niVLFAUz74IC640e1H767yaZ14lWcfz7Vm1yeAHGNARHFq3IcNOy3mZlY1+4dHJW0EAnMfwwPzkthnZtnFbtFucKTtA/SQMKjk6Bppbg5qXBKzU8njsw+QZha87aHks= ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:24:06.776 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpITC1mxHZ5j832LwvPCR1yWJlT+sGQ4zxvUoWarZU8AzXtUi4nvREiVv9Q52Wc+bZJYhc2FfVr+OdotcdnXOF59rlAe4mH3EOUHcMcNWagP4QaVYnYqxxLWF9+OdxXdS9Ko4y7pV9ndkRXkYBwXUju9yWf/hXcCuDlR97Z9j6TWLPDBRCKJbxHbUOzVlWDiEqoPwt9u68zzkWHdRSr57ClsW7hHBspRnre2ve90Q7yPHE9HdrJgqbxgPbFt88LcwT8L7oafy6KwiWR6f9iIYf9pRD9uN6YWf7581x5mJvkZVpLXHPf55suNmzjEjOXPsnFLz57rAEvOzm+/XpiTkq28y0BPFCocYtEz5xSyDA98G2ThanNChA/vdES+evTmq+niVLFAUz74IC640e1H767yaZ14lWcfz7Vm1yeAHGNARHFq3IcNOy3mZlY1+4dHJW0EAnMfwwPzkthnZtnFbtFucKTtA/SQMKjk6Bppbg5qXBKzU8njsw+QZha87aHks= ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:06.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpITC1mxHZ5j832LwvPCR1yWJlT+sGQ4zxvUoWarZU8AzXtUi4nvREiVv9Q52Wc+bZJYhc2FfVr+OdotcdnXOF59rlAe4mH3EOUHcMcNWagP4QaVYnYqxxLWF9+OdxXdS9Ko4y7pV9ndkRXkYBwXUju9yWf/hXcCuDlR97Z9j6TWLPDBRCKJbxHbUOzVlWDiEqoPwt9u68zzkWHdRSr57ClsW7hHBspRnre2ve90Q7yPHE9HdrJgqbxgPbFt88LcwT8L7oafy6KwiWR6f9iIYf9pRD9uN6YWf7581x5mJvkZVpLXHPf55suNmzjEjOXPsnFLz57rAEvOzm+/XpiTkq28y0BPFCocYtEz5xSyDA98G2ThanNChA/vdES+evTmq+niVLFAUz74IC640e1H767yaZ14lWcfz7Vm1yeAHGNARHFq3IcNOy3mZlY1+4dHJW0EAnMfwwPzkthnZtnFbtFucKTtA/SQMKjk6Bppbg5qXBKzU8njsw+QZha87aHks= ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T21:24:06.835 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpITC1mxHZ5j832LwvPCR1yWJlT+sGQ4zxvUoWarZU8AzXtUi4nvREiVv9Q52Wc+bZJYhc2FfVr+OdotcdnXOF59rlAe4mH3EOUHcMcNWagP4QaVYnYqxxLWF9+OdxXdS9Ko4y7pV9ndkRXkYBwXUju9yWf/hXcCuDlR97Z9j6TWLPDBRCKJbxHbUOzVlWDiEqoPwt9u68zzkWHdRSr57ClsW7hHBspRnre2ve90Q7yPHE9HdrJgqbxgPbFt88LcwT8L7oafy6KwiWR6f9iIYf9pRD9uN6YWf7581x5mJvkZVpLXHPf55suNmzjEjOXPsnFLz57rAEvOzm+/XpiTkq28y0BPFCocYtEz5xSyDA98G2ThanNChA/vdES+evTmq+niVLFAUz74IC640e1H767yaZ14lWcfz7Vm1yeAHGNARHFq3IcNOy3mZlY1+4dHJW0EAnMfwwPzkthnZtnFbtFucKTtA/SQMKjk6Bppbg5qXBKzU8njsw+QZha87aHks= ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:06.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T21:24:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3817799139' entity='client.admin' 2024-03-26T21:24:07.095 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:10.279 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T21:24:10.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T21:24:10.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2500335135' entity='client.admin' 2024-03-26T21:24:10.930 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:12.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:12.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:12.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:24:12.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:12.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:12.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:13.336 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi067 2024-03-26T21:24:13.336 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:24:13.336 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.conf 2024-03-26T21:24:13.354 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:24:13.354 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:13.411 INFO:tasks.cephadm:Adding host smithi067 to orchestrator... 2024-03-26T21:24:13.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch host add smithi067 2024-03-26T21:24:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:13.903 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:13.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:14.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:14 smithi055 ceph-mon[29406]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:14.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:14 smithi055 ceph-mon[29406]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:14.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:14 smithi055 ceph-mon[29406]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:24:16.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:15 smithi055 ceph-mon[29406]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:17.141 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:16 smithi055 ceph-mon[29406]: Deploying cephadm binary to smithi067 2024-03-26T21:24:18.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:17 smithi055 ceph-mon[29406]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:19.740 INFO:teuthology.orchestra.run.smithi055.stdout:Added host 'smithi067' with addr '172.21.15.67' 2024-03-26T21:24:20.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T21:24:20.575 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:20.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:20.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:20 smithi055 ceph-mon[29406]: Added host smithi067 2024-03-26T21:24:20.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:20.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:20 smithi055 ceph-mon[29406]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:20.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:21.980 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:22.214 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:24:22.215 INFO:teuthology.orchestra.run.smithi055.stdout:[{"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}] 2024-03-26T21:24:22.752 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T21:24:22.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T21:24:22.988 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:22 smithi055 ceph-mon[29406]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:22 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:24.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:23 smithi055 ceph-mon[29406]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:24:24.814 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2024-03-26T21:24:25.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:24 smithi055 ceph-mon[29406]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:24 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:24:25.421 INFO:tasks.cephadm:Adding mon.a on smithi055 2024-03-26T21:24:25.421 INFO:tasks.cephadm:Adding mon.c on smithi055 2024-03-26T21:24:25.421 INFO:tasks.cephadm:Adding mon.b on smithi067 2024-03-26T21:24:25.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b' 2024-03-26T21:24:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:24:25.702 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-26T21:24:26.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:25 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:27.425 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled mon update... 2024-03-26T21:24:28.003 DEBUG:teuthology.orchestra.run.smithi055:mon.c> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T21:24:28.006 DEBUG:teuthology.orchestra.run.smithi067:mon.b> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T21:24:28.008 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:24:28.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:28 smithi055 ceph-mon[29406]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:28 smithi055 ceph-mon[29406]: Saving service mon spec with placement smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b;count:3 2024-03-26T21:24:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:28 smithi055 ceph-mon[29406]: Deploying daemon mon.b on smithi067 2024-03-26T21:24:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:28 smithi055 ceph-mon[29406]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:24:31.230 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","modified":"2024-03-26T21:22:53.649249Z","created":"2024-03-26T21:22:53.649249Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T21:24:31.232 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 1 2024-03-26T21:24:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:30 smithi055 ceph-mon[29406]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:32.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:31 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/743719434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:32.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:24:32.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.row_cache: None 2024-03-26T21:24:32.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.wal_filter: None 2024-03-26T21:24:32.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:24:32.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.wal_compression: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:24:32.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:24:32.800 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:24:32.800 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:24:32.933 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:24:32.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:24:33.172 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:24:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_open_files: -1 2024-03-26T21:24:33.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Compression algorithms supported: 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kZSTD supported: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:24:33.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.merge_operator: 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_filter: None 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:24:33.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56410da35cc0) 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: index_type: 0 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: data_block_index_type: 0 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: index_shortening: 1 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: checksum: 4 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: no_block_cache: 0 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_cache: 0x56410da39090 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_cache_options: 2024-03-26T21:24:33.244 INFO:journalctl@ceph.mon.b.smithi067.stdout: capacity : 536870912 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: num_shard_bits : 4 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: strict_capacity_limit : 0 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_cache_compressed: (nil) 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: persistent_cache: (nil) 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_size: 4096 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_size_deviation: 10 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_restart_interval: 16 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: index_block_restart_interval: 1 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: metadata_block_size: 4096 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: partition_filters: 0 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: use_delta_encoding: 1 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: filter_policy: bloomfilter 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: whole_key_filtering: 1 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: verify_compression: 0 2024-03-26T21:24:33.245 INFO:journalctl@ceph.mon.b.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: format_version: 5 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: enable_index_compression: 1 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: block_align: 0 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: max_auto_readahead_size: 262144 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: prepopulate_block_cache: 0 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression: NoCompression 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:24:33.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.num_levels: 7 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:24:33.247 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:24:33.248 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:24:33.249 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:24:33.250 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:24:33.251 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.ttl: 2592000 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:24:33.252 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2078ccb0-5cbd-46b6-8b49-a3c39cd30792 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488272804372, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T21:24:33.253 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488272804955, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711488272, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2078ccb0-5cbd-46b6-8b49-a3c39cd30792", "db_session_id": "WWENZJTL5N2FQS4MZGZO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488272805068, "job": 1, "event": "recovery_finished"} 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56410db28000 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: DB pointer 0x56410db12000 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: ** DB Stats ** 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:24:33.254 INFO:journalctl@ceph.mon.b.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: ** Compaction Stats [default] ** 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: ** Compaction Stats [default] ** 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:24:33.255 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Block cache BinnedLRUCache@0x56410da39090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: using public_addr v2:172.21.15.67:0/0 -> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2024-03-26T21:24:33.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: starting mon.b rank -1 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(???) e0 preinit fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: e1 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: legacy client fscid: -1 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout: No filesystems configured 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T21:24:33.257 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:24:33.258 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.259 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Saving service mon spec with placement smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b;count:3 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: Deploying daemon mon.b on smithi067 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/743719434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 ceph-mon[33346]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T21:24:33.260 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 bash[33326]: 84744af93b1ff22e9abb3f2ccbc05e17970a7f02e9d172140304ca1ed22e8f7d 2024-03-26T21:24:33.261 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:32 smithi067 systemd[1]: Started Ceph mon.b for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: mon.a calling monitor election 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: mon.b calling monitor election 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: monmap epoch 2 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: last_changed 2024-03-26T21:24:32.828205+0000 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: min_mon_release 19 (squid) 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: election_strategy: 1 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: fsmap 2024-03-26T21:24:38.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: mgrmap e13: y(active, since 40s) 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: overall HEALTH_OK 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:38.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: mon.a calling monitor election 2024-03-26T21:24:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: mon.b calling monitor election 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: monmap epoch 2 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: last_changed 2024-03-26T21:24:32.828205+0000 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: min_mon_release 19 (squid) 2024-03-26T21:24:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: election_strategy: 1 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: fsmap 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: mgrmap e13: y(active, since 40s) 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: overall HEALTH_OK 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:38.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:24:38.416 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","modified":"2024-03-26T21:24:32.828205Z","created":"2024-03-26T21:22:53.649249Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T21:24:38.418 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 2 2024-03-26T21:24:38.951 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:38 smithi067 ceph-mon[33346]: Deploying daemon mon.c on smithi055 2024-03-26T21:24:38.952 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/963004887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:38.952 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:38 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:38 smithi055 ceph-mon[29406]: Deploying daemon mon.c on smithi055 2024-03-26T21:24:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/963004887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:38 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:39.979 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T21:24:39.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T21:24:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:24:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:39 smithi067 ceph-mon[33346]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:39 smithi055 ceph-mon[29406]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:40 smithi055 systemd[1]: Starting Ceph mon.c for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:24:41.337 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 podman[40567]: 2024-03-26 21:24:40.983779396 +0000 UTC m=+0.017554473 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:24:41.337 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 podman[40567]: 2024-03-26 21:24:41.219567921 +0000 UTC m=+0.253342996 container create dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 podman[40567]: 2024-03-26 21:24:41.43009651 +0000 UTC m=+0.463871584 container init dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 podman[40567]: 2024-03-26 21:24:41.433054068 +0000 UTC m=+0.466829142 container start dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, ceph=True) 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev), process ceph-mon, pid 2 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pidfile_write: ignore empty --pid-file 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: load: jerasure load: lrc 2024-03-26T21:24:41.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: RocksDB version: 7.9.2 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Git sha 0 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Compile date 2024-03-26 11:42:14 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: DB SUMMARY 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: DB Session ID: BY1CU4P4CXJBHQDDHAWS 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: CURRENT file: CURRENT 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.error_if_exists: 0 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.create_if_missing: 0 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.paranoid_checks: 1 2024-03-26T21:24:41.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.env: 0x55ff37bf5c60 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.info_log: 0x55ff3868fae0 2024-03-26T21:24:41.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.statistics: (nil) 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.use_fsync: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_log_file_size: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_fallocate: 1 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T21:24:41.820 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.use_direct_reads: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.db_log_dir: 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.wal_dir: 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.write_buffer_manager: 0x55ff386601e0 2024-03-26T21:24:41.821 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.unordered_write: 0 2024-03-26T21:24:41.822 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.row_cache: None 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.wal_filter: None 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.two_write_queues: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.wal_compression: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.atomic_flush: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T21:24:41.823 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.log_readahead_size: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_background_jobs: 2 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_background_compactions: -1 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_subcompactions: 1 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T21:24:41.824 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_open_files: -1 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_background_flushes: -1 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Compression algorithms supported: 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kZSTD supported: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kXpressCompression supported: 0 2024-03-26T21:24:41.825 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kZlibCompression supported: 1 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kSnappyCompression supported: 1 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kLZ4Compression supported: 1 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: kBZip2Compression supported: 0 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.merge_operator: 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_filter: None 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T21:24:41.826 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ff3868fcc0) 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: cache_index_and_filter_blocks: 1 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: pin_top_level_index_and_filter: 1 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: index_type: 0 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: data_block_index_type: 0 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: index_shortening: 1 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: checksum: 4 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: no_block_cache: 0 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_cache: 0x55ff38693090 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_cache_name: BinnedLRUCache 2024-03-26T21:24:41.827 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_cache_options: 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: capacity : 536870912 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: num_shard_bits : 4 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: strict_capacity_limit : 0 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: high_pri_pool_ratio: 0.000 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_cache_compressed: (nil) 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: persistent_cache: (nil) 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_size: 4096 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_size_deviation: 10 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_restart_interval: 16 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: index_block_restart_interval: 1 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: metadata_block_size: 4096 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: partition_filters: 0 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: use_delta_encoding: 1 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: filter_policy: bloomfilter 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: whole_key_filtering: 1 2024-03-26T21:24:41.828 INFO:journalctl@ceph.mon.c.smithi055.stdout: verify_compression: 0 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: read_amp_bytes_per_bit: 0 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: format_version: 5 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: enable_index_compression: 1 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: block_align: 0 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: max_auto_readahead_size: 262144 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: prepopulate_block_cache: 0 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: initial_auto_readahead_size: 8192 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression: NoCompression 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.num_levels: 7 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T21:24:41.829 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T21:24:41.830 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T21:24:41.831 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T21:24:41.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.inplace_update_support: 0 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.bloom_locality: 0 2024-03-26T21:24:41.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.max_successive_merges: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.ttl: 2592000 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enable_blob_files: false 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.min_blob_size: 0 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T21:24:41.834 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bcf1d5ce-43d0-46ef-b75a-924201a1421b 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488281462995, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488281463532, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711488281, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bcf1d5ce-43d0-46ef-b75a-924201a1421b", "db_session_id": "BY1CU4P4CXJBHQDDHAWS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711488281463612, "job": 1, "event": "recovery_finished"} 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ff38782000 2024-03-26T21:24:41.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: DB pointer 0x55ff3876c000 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: ** DB Stats ** 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:41.836 INFO:journalctl@ceph.mon.c.smithi055.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: ** Compaction Stats [default] ** 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T21:24:41.837 INFO:journalctl@ceph.mon.c.smithi055.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: Block cache BinnedLRUCache@0x55ff38693090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: using public_addrv [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: starting mon.c rank -1 at public addrs [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] at bind addrs [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(???) e0 preinit fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T21:24:41.838 INFO:journalctl@ceph.mon.c.smithi055.stdout: e1 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout: legacy client fscid: -1 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout: 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout: No filesystems configured 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:41.839 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1624311021' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.840 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.841 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Saving service mon spec with placement smithi055:172.21.15.55=a;smithi055:[v2:172.21.15.55:3301,v1:172.21.15.55:6790]=c;smithi067:172.21.15.67=b;count:3 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Deploying daemon mon.b on smithi067 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/743719434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:41.842 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.a calling monitor election 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.b calling monitor election 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: monmap epoch 2 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:41.843 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: last_changed 2024-03-26T21:24:32.828205+0000 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: min_mon_release 19 (squid) 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: election_strategy: 1 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: fsmap 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mgrmap e13: y(active, since 40s) 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: overall HEALTH_OK 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: Deploying daemon mon.c on smithi055 2024-03-26T21:24:41.844 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/963004887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:41.845 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:41.845 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:41.845 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 ceph-mon[40587]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T21:24:41.845 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 bash[40567]: dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 2024-03-26T21:24:41.845 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:41 smithi055 systemd[1]: Started Ceph mon.c for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: mon.a calling monitor election 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: mon.b calling monitor election 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: mon.c calling monitor election 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: monmap epoch 3 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: last_changed 2024-03-26T21:24:41.488809+0000 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: min_mon_release 19 (squid) 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: election_strategy: 1 2024-03-26T21:24:46.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: 2: [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] mon.c 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: fsmap 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: mgrmap e13: y(active, since 48s) 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: overall HEALTH_OK 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: mon.a calling monitor election 2024-03-26T21:24:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: mon.b calling monitor election 2024-03-26T21:24:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: mon.c calling monitor election 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: monmap epoch 3 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: last_changed 2024-03-26T21:24:41.488809+0000 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:47.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: min_mon_release 19 (squid) 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: election_strategy: 1 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: 2: [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] mon.c 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: fsmap 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: mgrmap e13: y(active, since 48s) 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: overall HEALTH_OK 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: mon.a calling monitor election 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: mon.b calling monitor election 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: mon.c calling monitor election 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: monmap epoch 3 2024-03-26T21:24:47.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: last_changed 2024-03-26T21:24:41.488809+0000 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: created 2024-03-26T21:22:53.649249+0000 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: min_mon_release 19 (squid) 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: election_strategy: 1 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: 0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: 1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: 2: [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] mon.c 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: fsmap 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: mgrmap e13: y(active, since 48s) 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: overall HEALTH_OK 2024-03-26T21:24:47.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.071 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.071 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.071 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:47.813 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-26T21:24:47.813 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","modified":"2024-03-26T21:24:41.488809Z","created":"2024-03-26T21:22:53.649249Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3301","nonce":0},{"type":"v1","addr":"172.21.15.55:6790","nonce":0}]},"addr":"172.21.15.55:6790/0","public_addr":"172.21.15.55:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T21:24:47.813 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:47.813 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.815 INFO:teuthology.orchestra.run.smithi067.stderr:dumped monmap epoch 3 2024-03-26T21:24:47.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:47.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:47.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:47.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:24:48.519 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T21:24:48.519 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T21:24:48.774 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/425733966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/425733966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:49.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/425733966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:49.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:49.962 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:24:49.962 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:49.962 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[40587]: Reconfiguring daemon mon.a on smithi055 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:24:49.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:49 smithi055 ceph-mon[29406]: Reconfiguring daemon mon.a on smithi055 2024-03-26T21:24:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:24:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:24:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:24:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T21:24:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:49 smithi067 ceph-mon[33346]: Reconfiguring daemon mon.a on smithi055 2024-03-26T21:24:50.951 INFO:teuthology.orchestra.run.smithi055.stdout:# minimal ceph.conf for e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:50.951 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-03-26T21:24:50.951 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:24:50.951 INFO:teuthology.orchestra.run.smithi055.stdout: mon_host = [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.55:3301/0,v1:172.21.15.55:6790/0] 2024-03-26T21:24:51.160 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:50 smithi055 ceph-mon[29406]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:51.160 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:50 smithi055 ceph-mon[40587]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:50 smithi067 ceph-mon[33346]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:51.723 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T21:24:51.723 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:51.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T21:24:51.753 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:51.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:51.820 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:24:51.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T21:24:51.854 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:24:51.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:24:51.924 INFO:tasks.cephadm:Adding mgr.y on smithi055 2024-03-26T21:24:51.924 INFO:tasks.cephadm:Adding mgr.x on smithi067 2024-03-26T21:24:51.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi055=y;smithi067=x' 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/537706539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:51.972 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/537706539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:51.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:51.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:51.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:51.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:51.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:52.203 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:24:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/537706539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: Reconfiguring daemon mon.c on smithi055 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:52.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: Reconfiguring daemon mon.c on smithi055 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:52.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: Reconfiguring mon.c (monmap changed)... 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: Reconfiguring daemon mon.c on smithi055 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T21:24:53.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:53 smithi067 ceph-mon[33346]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:24:53.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:53 smithi067 ceph-mon[33346]: Reconfiguring daemon mon.b on smithi067 2024-03-26T21:24:54.082 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled mgr update... 2024-03-26T21:24:54.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:53 smithi055 ceph-mon[29406]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:24:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:53 smithi055 ceph-mon[29406]: Reconfiguring daemon mon.b on smithi067 2024-03-26T21:24:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:53 smithi055 ceph-mon[40587]: Reconfiguring mon.b (monmap changed)... 2024-03-26T21:24:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:53 smithi055 ceph-mon[40587]: Reconfiguring daemon mon.b on smithi067 2024-03-26T21:24:54.743 DEBUG:teuthology.orchestra.run.smithi067:mgr.x> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T21:24:54.745 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T21:24:54.745 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:24:54.746 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T21:24:54.764 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T21:24:54.765 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-03-26T21:24:54.822 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T21:24:54.822 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:54.822 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T21:24:54.822 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:54.822 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:54.823 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 21:24:40.810120469 +0000 2024-03-26T21:24:54.823 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 21:17:12.160667122 +0000 2024-03-26T21:24:54.823 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 21:17:12.160667122 +0000 2024-03-26T21:24:54.823 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 21:17:12.160667122 +0000 2024-03-26T21:24:54.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T21:24:54.893 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T21:24:54.893 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T21:24:54.893 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.00029932 s, 1.7 MB/s 2024-03-26T21:24:54.894 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T21:24:54.952 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-03-26T21:24:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:54 smithi067 ceph-mon[33346]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:54 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 21:24:40.810120469 +0000 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 21:17:12.517659515 +0000 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 21:17:12.517659515 +0000 2024-03-26T21:24:55.009 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 21:17:12.517659515 +0000 2024-03-26T21:24:55.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T21:24:55.079 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T21:24:55.079 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T21:24:55.079 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000317034 s, 1.6 MB/s 2024-03-26T21:24:55.080 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T21:24:55.139 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 21:24:40.811120448 +0000 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 21:17:12.848652462 +0000 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 21:17:12.848652462 +0000 2024-03-26T21:24:55.196 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 21:17:12.848652462 +0000 2024-03-26T21:24:55.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T21:24:55.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:54 smithi055 ceph-mon[40587]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:55.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:54 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:54 smithi055 ceph-mon[29406]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:55.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:54 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.264 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T21:24:55.264 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T21:24:55.264 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000297035 s, 1.7 MB/s 2024-03-26T21:24:55.265 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T21:24:55.322 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-03-26T21:24:55.377 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T21:24:55.377 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.377 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T21:24:55.377 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.377 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.378 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-26 21:24:40.811120448 +0000 2024-03-26T21:24:55.378 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-26 21:17:13.191645153 +0000 2024-03-26T21:24:55.378 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-26 21:17:13.191645153 +0000 2024-03-26T21:24:55.378 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-26 21:17:13.191645153 +0000 2024-03-26T21:24:55.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T21:24:55.443 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-26T21:24:55.444 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-26T21:24:55.444 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000292264 s, 1.8 MB/s 2024-03-26T21:24:55.445 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T21:24:55.501 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:24:55.502 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T21:24:55.517 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T21:24:55.517 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-26 21:24:32.248684248 +0000 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-26 21:19:16.079235166 +0000 2024-03-26T21:24:55.571 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-26 21:19:16.079235166 +0000 2024-03-26T21:24:55.572 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-26 21:19:16.079235166 +0000 2024-03-26T21:24:55.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T21:24:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-26T21:24:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-26T21:24:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000249518 s, 2.1 MB/s 2024-03-26T21:24:55.640 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T21:24:55.697 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-03-26T21:24:55.751 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-26 21:24:32.249684226 +0000 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-26 21:19:16.418227492 +0000 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-26 21:19:16.418227492 +0000 2024-03-26T21:24:55.752 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-26 21:19:16.418227492 +0000 2024-03-26T21:24:55.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T21:24:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-26T21:24:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-26T21:24:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000189612 s, 2.7 MB/s 2024-03-26T21:24:55.818 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T21:24:55.874 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi055=y;smithi067=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: Saving service mgr spec with placement smithi055=y;smithi067=x;count:2 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:55.913 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: Deploying daemon mgr.x on smithi067 2024-03-26T21:24:55.914 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:55 smithi067 ceph-mon[33346]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:55.927 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 697 Links: 1 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-26 21:24:32.249684226 +0000 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-26 21:19:16.762219705 +0000 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-26 21:19:16.762219705 +0000 2024-03-26T21:24:55.928 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-26 21:19:16.762219705 +0000 2024-03-26T21:24:55.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T21:24:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-26T21:24:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-26T21:24:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000187333 s, 2.7 MB/s 2024-03-26T21:24:55.995 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T21:24:56.053 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-03-26T21:24:56.109 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T21:24:56.109 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 716 Links: 1 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-26 21:24:32.249684226 +0000 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-26 21:19:17.109211849 +0000 2024-03-26T21:24:56.110 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-26 21:19:17.109211849 +0000 2024-03-26T21:24:56.111 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-26 21:19:17.109211849 +0000 2024-03-26T21:24:56.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T21:24:56.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-26T21:24:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-26T21:24:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000171368 s, 3.0 MB/s 2024-03-26T21:24:56.176 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T21:24:56.231 INFO:tasks.cephadm:Deploying osd.0 on smithi055 with /dev/vg_nvme/lv_4... 2024-03-26T21:24:56.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi055=y;smithi067=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: Saving service mgr spec with placement smithi055=y;smithi067=x;count:2 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:56.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: Deploying daemon mgr.x on smithi067 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[40587]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi055=y;smithi067=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: Saving service mgr spec with placement smithi055=y;smithi067=x;count:2 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:24:56.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:24:56.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: Deploying daemon mgr.x on smithi067 2024-03-26T21:24:56.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:55 smithi055 ceph-mon[29406]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:56.484 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:57.739 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:57 smithi067 systemd[1]: Started Ceph mgr.x for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00928882 s, 1.1 GB/s 2024-03-26T21:24:58.342 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-03-26T21:24:58.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:58.129+0000 7f7f25463200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:24:58.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:58.226+0000 7f7f25463200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:24:58.900 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:24:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:58.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_4 2024-03-26T21:24:58.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:58.921+0000 7f7f25463200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:24:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:24:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:24:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:24:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:24:59.214 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:24:59.347 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.017+0000 7f7f25463200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:24:59.347 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.167+0000 7f7f25463200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:24:59.347 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.267+0000 7f7f25463200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:24:59.347 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.353+0000 7f7f25463200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:24:59.913 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.622+0000 7f7f25463200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:24:59.913 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.735+0000 7f7f25463200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:24:59.913 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.823+0000 7f7f25463200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:24:59.913 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:24:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:24:59.919+0000 7f7f25463200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:25:00.204 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:00.097+0000 7f7f25463200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:25:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: Reconfiguring daemon mgr.y on smithi055 2024-03-26T21:25:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:00 smithi067 ceph-mon[33346]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:00.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:00.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: Reconfiguring daemon mgr.y on smithi055 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[40587]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T21:25:00.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T21:25:00.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T21:25:00.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:00.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: Reconfiguring daemon mgr.y on smithi055 2024-03-26T21:25:00.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:00 smithi055 ceph-mon[29406]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:01.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:01.097+0000 7f7f25463200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:25:01.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:01.230+0000 7f7f25463200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:25:01.861 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:01.574+0000 7f7f25463200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:25:01.861 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:01.665+0000 7f7f25463200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:25:01.861 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:01.869+0000 7f7f25463200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:25:02.148 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:02.059+0000 7f7f25463200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:02.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:02.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:02.351+0000 7f7f25463200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:25:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:02.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:02.613+0000 7f7f25463200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:25:02.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:02.855+0000 7f7f25463200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:25:03.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:03 smithi067 ceph-mon[33346]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:03.990 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:03.677+0000 7f7f25463200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:25:04.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:03 smithi055 ceph-mon[29406]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:04.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:03 smithi055 ceph-mon[40587]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:04.571 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:04.258+0000 7f7f25463200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:25:04.571 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:04.357+0000 7f7f25463200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:25:04.572 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:25:04.578+0000 7f7f25463200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:25:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3924208813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]': finished 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: Standby manager daemon x started 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:25:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:04 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3924208813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]': finished 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: Standby manager daemon x started 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:25:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3924208813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae"}]': finished 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: Standby manager daemon x started 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:25:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:04 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2565236120' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:25:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:05 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4043593831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:05 smithi067 ceph-mon[33346]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T21:25:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:25:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:05 smithi067 ceph-mon[33346]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4043593831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[29406]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T21:25:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:25:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[29406]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4043593831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[40587]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T21:25:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:25:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:05 smithi055 ceph-mon[40587]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:07.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:07 smithi055 ceph-mon[29406]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:07.958 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:07 smithi055 ceph-mon[40587]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:07 smithi067 ceph-mon[33346]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:09 smithi055 ceph-mon[29406]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:09 smithi055 ceph-mon[40587]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:09 smithi067 ceph-mon[33346]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:10 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:25:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:10 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:10 smithi067 ceph-mon[33346]: Deploying daemon osd.0 on smithi055 2024-03-26T21:25:11.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:25:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[29406]: Deploying daemon osd.0 on smithi055 2024-03-26T21:25:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T21:25:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:10 smithi055 ceph-mon[40587]: Deploying daemon osd.0 on smithi055 2024-03-26T21:25:12.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:11 smithi055 ceph-mon[29406]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:11 smithi055 ceph-mon[40587]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:11 smithi067 ceph-mon[33346]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:13 smithi067 ceph-mon[33346]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:14.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:13 smithi055 ceph-mon[29406]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:14.314 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:13 smithi055 ceph-mon[40587]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:15 smithi067 ceph-mon[33346]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:16.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:15 smithi055 ceph-mon[29406]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:16.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:15 smithi055 ceph-mon[40587]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:17 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:17 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:17 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:17.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:17 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:18 smithi067 ceph-mon[33346]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:18.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:18 smithi055 ceph-mon[29406]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:18 smithi055 ceph-mon[40587]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:19 smithi055 ceph-mon[29406]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:25:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:19 smithi055 ceph-mon[40587]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:25:19.679 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 0 on host 'smithi055' 2024-03-26T21:25:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:19 smithi067 ceph-mon[33346]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T21:25:20.262 DEBUG:teuthology.orchestra.run.smithi055:osd.0> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T21:25:20.264 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/vg_nvme/lv_3... 2024-03-26T21:25:20.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T21:25:20.486 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:25:20.486 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:25:20.486 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:20.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[29406]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:25:20.575 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:25:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:25:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:20 smithi067 ceph-mon[33346]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:20.813 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-mon[40587]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T21:25:21.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0[44589]: 2024-03-26T21:25:20.842+0000 7f2985cd2700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T21:25:21.066 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:25:20 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0[44589]: 2024-03-26T21:25:20.848+0000 7f297dabd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' 2024-03-26T21:25:21.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='osd.0 v2:172.21.15.55:6801/317381212' entity='osd.0' 2024-03-26T21:25:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0224279 s, 468 MB/s 2024-03-26T21:25:22.571 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-03-26T21:25:23.128 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: osd.0 v2:172.21.15.55:6801/317381212 boot 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: osd.0 v2:172.21.15.55:6801/317381212 boot 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:25:23.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:22 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:25:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:25:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T21:25:23.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: osd.0 v2:172.21.15.55:6801/317381212 boot 2024-03-26T21:25:23.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T21:25:23.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:22 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:25:23.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_3 2024-03-26T21:25:24.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:23 smithi055 ceph-mon[29406]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:25:24.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:23 smithi055 ceph-mon[29406]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:23 smithi055 ceph-mon[40587]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:25:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:23 smithi055 ceph-mon[40587]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:23 smithi067 ceph-mon[33346]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T21:25:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:23 smithi067 ceph-mon[33346]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:24.283 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi055 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:26.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:26.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:26.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:26 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi055 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi055 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 16069M 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:27.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:27.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:26 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:27 smithi067 ceph-mon[33346]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[40587]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[29406]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:27.993 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:27.994 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:27.994 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:28 smithi067 ceph-mon[33346]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:29.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:28 smithi055 ceph-mon[29406]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:28 smithi055 ceph-mon[40587]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:29.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]: dispatch 2024-03-26T21:25:29.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]': finished 2024-03-26T21:25:29.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:29 smithi067 ceph-mon[33346]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:25:29.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:29 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]: dispatch 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]': finished 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[40587]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]: dispatch 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1881006545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db"}]': finished 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[29406]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T21:25:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:29 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:30 smithi067 ceph-mon[33346]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:30 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3670473620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:30 smithi055 ceph-mon[40587]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:30 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3670473620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:30 smithi055 ceph-mon[29406]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:30 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3670473620' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:31 smithi067 ceph-mon[33346]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:32.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:31 smithi055 ceph-mon[29406]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:31 smithi055 ceph-mon[40587]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:33 smithi055 ceph-mon[40587]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:33 smithi055 ceph-mon[29406]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:33 smithi067 ceph-mon[33346]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:35.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:25:35.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:35.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:25:35.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:35.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T21:25:35.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:36.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:36 smithi055 ceph-mon[40587]: Deploying daemon osd.1 on smithi055 2024-03-26T21:25:36.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:36 smithi055 ceph-mon[40587]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:36 smithi055 ceph-mon[29406]: Deploying daemon osd.1 on smithi055 2024-03-26T21:25:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:36 smithi055 ceph-mon[29406]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:36 smithi067 ceph-mon[33346]: Deploying daemon osd.1 on smithi055 2024-03-26T21:25:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:36 smithi067 ceph-mon[33346]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:37 smithi055 ceph-mon[40587]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:37 smithi055 ceph-mon[29406]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:37 smithi067 ceph-mon[33346]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:39 smithi055 ceph-mon[29406]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:40.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:39 smithi055 ceph-mon[40587]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:39 smithi067 ceph-mon[33346]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[29406]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:42.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:42 smithi055 ceph-mon[40587]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:42 smithi067 ceph-mon[33346]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:44.056 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:43 smithi055 ceph-mon[29406]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:44.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:43 smithi055 ceph-mon[40587]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:43 smithi067 ceph-mon[33346]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:45.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:45.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:44 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T21:25:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:44 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:45.285 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2024-03-26T21:25:45.947 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T21:25:45.949 INFO:tasks.cephadm:Deploying osd.2 on smithi055 with /dev/vg_nvme/lv_2... 2024-03-26T21:25:45.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[29406]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:46.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:45 smithi055 ceph-mon[40587]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T21:25:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T21:25:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='osd.1 v2:172.21.15.55:6805/3733554168' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:25:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:45 smithi067 ceph-mon[33346]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:46.566 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1[50597]: 2024-03-26T21:25:46.409+0000 7f3c15a97700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T21:25:46.566 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1[50597]: 2024-03-26T21:25:46.412+0000 7f3c0d882700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:25:46.701 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[29406]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[29406]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[29406]: from='osd.1 ' entity='osd.1' 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[40587]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[40587]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.075 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:46 smithi055 ceph-mon[40587]: from='osd.1 ' entity='osd.1' 2024-03-26T21:25:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:46 smithi067 ceph-mon[33346]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:25:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:46 smithi067 ceph-mon[33346]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T21:25:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:46 smithi067 ceph-mon[33346]: from='osd.1 ' entity='osd.1' 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: osd.1 v2:172.21.15.55:6805/3733554168 boot 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[29406]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: osd.1 v2:172.21.15.55:6805/3733554168 boot 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:47 smithi055 ceph-mon[40587]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: osd.1 v2:172.21.15.55:6805/3733554168 boot 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T21:25:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:25:48.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:47 smithi067 ceph-mon[33346]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T21:25:49.593 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T21:25:49.593 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T21:25:49.593 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-03-26T21:25:49.593 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-03-26T21:25:49.593 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107583 s, 975 MB/s 2024-03-26T21:25:49.594 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-03-26T21:25:50.222 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:49 smithi055 ceph-mon[29406]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:25:50.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:49 smithi055 ceph-mon[29406]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:50.223 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:49 smithi055 ceph-mon[40587]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:25:50.223 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:49 smithi055 ceph-mon[40587]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:49 smithi067 ceph-mon[33346]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T21:25:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:49 smithi067 ceph-mon[33346]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:50.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_2 2024-03-26T21:25:50.933 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi055 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 8034M 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.027 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[29406]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi055 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 8034M 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.028 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:51 smithi055 ceph-mon[40587]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi055 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 8034M 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:25:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:51 smithi067 ceph-mon[33346]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:53.700 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:25:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:25:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:25:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:54 smithi055 ceph-mon[40587]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:54 smithi055 ceph-mon[40587]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:54.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:54 smithi055 ceph-mon[29406]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:54.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:54 smithi055 ceph-mon[29406]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:54 smithi067 ceph-mon[33346]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:25:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:54 smithi067 ceph-mon[33346]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/940173664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]': finished 2024-03-26T21:25:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:55 smithi067 ceph-mon[33346]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:25:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:55 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:25:56.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/940173664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]': finished 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[29406]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/940173664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]: dispatch 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b"}]': finished 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[40587]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T21:25:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:55 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:25:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:56 smithi067 ceph-mon[33346]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/592151245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:56 smithi055 ceph-mon[40587]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/592151245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:56 smithi055 ceph-mon[29406]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/592151245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:25:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:57 smithi067 ceph-mon[33346]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:57 smithi055 ceph-mon[40587]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:25:58.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:57 smithi055 ceph-mon[29406]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:00.220 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:25:59 smithi055 ceph-mon[29406]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:00.221 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:25:59 smithi055 ceph-mon[40587]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:25:59 smithi067 ceph-mon[33346]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:02 smithi067 ceph-mon[33346]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:26:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:02.806 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[29406]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:02.806 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:26:02.806 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:02.807 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[40587]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:02.807 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T21:26:02.807 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:03 smithi067 ceph-mon[33346]: Deploying daemon osd.2 on smithi055 2024-03-26T21:26:03.761 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:03 smithi055 ceph-mon[29406]: Deploying daemon osd.2 on smithi055 2024-03-26T21:26:03.761 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:03 smithi055 ceph-mon[40587]: Deploying daemon osd.2 on smithi055 2024-03-26T21:26:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:04 smithi067 ceph-mon[33346]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:04.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:04 smithi055 ceph-mon[29406]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:04.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:04 smithi055 ceph-mon[40587]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:06.156 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:05 smithi055 ceph-mon[29406]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:06.156 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:05 smithi055 ceph-mon[40587]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:05 smithi067 ceph-mon[33346]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[29406]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:09.043 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.044 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:08 smithi055 ceph-mon[40587]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:08 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:08 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:08 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:08 smithi067 ceph-mon[33346]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:10.044 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:09 smithi055 ceph-mon[29406]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:09 smithi055 ceph-mon[40587]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:09 smithi067 ceph-mon[33346]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:11.399 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 2 on host 'smithi055' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T21:26:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:11.974 DEBUG:teuthology.orchestra.run.smithi055:osd.2> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T21:26:11.976 INFO:tasks.cephadm:Deploying osd.3 on smithi055 with /dev/vg_nvme/lv_1... 2024-03-26T21:26:11.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[29406]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[29406]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[29406]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[29406]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[40587]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[40587]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[40587]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-mon[40587]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:12 smithi067 ceph-mon[33346]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T21:26:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:12 smithi067 ceph-mon[33346]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T21:26:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:12 smithi067 ceph-mon[33346]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:12 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:12 smithi067 ceph-mon[33346]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:13.096 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2[56336]: 2024-03-26T21:26:12.846+0000 7f8c45bac700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T21:26:13.096 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2[56336]: 2024-03-26T21:26:12.852+0000 7f8c3d997700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:26:13.581 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.705 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' 2024-03-26T21:26:13.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:13 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: from='osd.2 v2:172.21.15.55:6809/3059242923' entity='osd.2' 2024-03-26T21:26:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:13 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:15.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: osd.2 v2:172.21.15.55:6809/3059242923 boot 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: osd.2 v2:172.21.15.55:6809/3059242923 boot 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:26:15.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:15.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:14 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: osd.2 v2:172.21.15.55:6809/3059242923 boot 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:26:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:14 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi055 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 5356M 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi055 2024-03-26T21:26:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 5356M 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:16.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:15 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T21:26:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi055 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 5356M 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:15 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00940891 s, 1.1 GB/s 2024-03-26T21:26:16.271 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-03-26T21:26:16.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60069]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60069]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60069]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60069]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60072]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60072]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60072]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60072]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60075]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60075]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60075]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60078]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60078]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60078]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60078]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60081]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60088]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60088]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60088]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.816 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60088]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60091]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60091]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60091]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60094]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60094]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60094]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60097]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60097]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60097]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60097]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60100]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60100]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 sudo[60100]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35897]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35897]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35900]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35900]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T21:26:16.864 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35900]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T21:26:16.865 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 sudo[35900]: pam_unix(sudo:session): session closed for user root 2024-03-26T21:26:16.865 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:26:16.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_1 2024-03-26T21:26:17.184 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:16 smithi067 ceph-mon[33346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.305 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:26:17.305 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.305 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.305 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.305 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[40587]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:26:17.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:16 smithi055 ceph-mon[29406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[29406]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[29406]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[29406]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[40587]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[40587]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:26:18.206 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:17 smithi055 ceph-mon[40587]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:17 smithi067 ceph-mon[33346]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T21:26:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:17 smithi067 ceph-mon[33346]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T21:26:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:17 smithi067 ceph-mon[33346]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:18 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:26:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:18 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:26:19.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:18 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:26:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:18 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:26:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:18 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:26:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:18 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:26:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:19 smithi067 ceph-mon[33346]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:19 smithi067 ceph-mon[33346]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[29406]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[29406]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[40587]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:19 smithi055 ceph-mon[40587]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]: dispatch 2024-03-26T21:26:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]': finished 2024-03-26T21:26:22.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:26:22.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:22.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:22.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3487935553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]: dispatch 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]': finished 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3487935553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]: dispatch 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3156386145' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b"}]': finished 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T21:26:22.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:22.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:22.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3487935553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:23 smithi067 ceph-mon[33346]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:23 smithi055 ceph-mon[29406]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:23 smithi055 ceph-mon[40587]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:26.140 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:25 smithi055 ceph-mon[29406]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:26.140 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:25 smithi055 ceph-mon[40587]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:25 smithi067 ceph-mon[33346]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:27 smithi067 ceph-mon[33346]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:28.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:27 smithi055 ceph-mon[29406]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:27 smithi055 ceph-mon[40587]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[40587]: Deploying daemon osd.3 on smithi055 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:29.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:28 smithi055 ceph-mon[29406]: Deploying daemon osd.3 on smithi055 2024-03-26T21:26:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:28 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T21:26:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:28 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:28 smithi067 ceph-mon[33346]: Deploying daemon osd.3 on smithi055 2024-03-26T21:26:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:29 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:26:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:29 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:26:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:29 smithi067 ceph-mon[33346]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:26:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:29 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:29 smithi067 ceph-mon[33346]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:30.264 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[29406]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[29406]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[40587]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:30.265 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:29 smithi055 ceph-mon[40587]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:31.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:31 smithi055 ceph-mon[29406]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:31 smithi067 ceph-mon[33346]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:32.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:31 smithi055 ceph-mon[40587]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:34.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:34.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:34 smithi067 ceph-mon[33346]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:35.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:35.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[29406]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:34 smithi055 ceph-mon[40587]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:36.218 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:35 smithi055 ceph-mon[29406]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:36.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:35 smithi055 ceph-mon[40587]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:35 smithi067 ceph-mon[33346]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:37.512 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 3 on host 'smithi055' 2024-03-26T21:26:37.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:26:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T21:26:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:37.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:38.159 DEBUG:teuthology.orchestra.run.smithi055:osd.3> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T21:26:38.161 INFO:tasks.cephadm:Deploying osd.4 on smithi067 with /dev/vg_nvme/lv_4... 2024-03-26T21:26:38.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T21:26:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:26:38.755 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:38 smithi067 ceph-mon[33346]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:26:38.755 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:38 smithi067 ceph-mon[33346]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:26:38.755 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:38 smithi067 ceph-mon[33346]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:38.755 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:38 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:38.755 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:38 smithi067 ceph-mon[33346]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[29406]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[29406]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[29406]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[29406]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[40587]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[40587]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[40587]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-mon[40587]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:39.296 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:26:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:26:38.993+0000 7fbbd9a8b700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T21:26:39.296 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:26:39.000+0000 7fbbd1876700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:39.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T21:26:39.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' 2024-03-26T21:26:39.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:39 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.883 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-03-26T21:26:39.883 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T21:26:39.883 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.883 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:39.884 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: from='osd.3 v2:172.21.15.55:6813/3851321468' entity='osd.3' 2024-03-26T21:26:39.884 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:39 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111896 s, 937 MB/s 2024-03-26T21:26:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: osd.3 v2:172.21.15.55:6813/3851321468 boot 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:40.796 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: osd.3 v2:172.21.15.55:6813/3851321468 boot 2024-03-26T21:26:40.797 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:26:40.797 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:40 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:26:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:26:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T21:26:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: osd.3 v2:172.21.15.55:6813/3851321468 boot 2024-03-26T21:26:40.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T21:26:40.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:40 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:26:41.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-03-26T21:26:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi055 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 4017M 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:42.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:42.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T21:26:42.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:42 smithi067 ceph-mon[33346]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi055 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 4017M 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[29406]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi055 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 4017M 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:26:42.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:42.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T21:26:42.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:42 smithi055 ceph-mon[40587]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:43 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:43 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:43 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:26:44.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:43 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:44 smithi067 ceph-mon[33346]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:44 smithi067 ceph-mon[33346]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:45.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:44 smithi055 ceph-mon[29406]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:45.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:44 smithi055 ceph-mon[29406]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:45.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:44 smithi055 ceph-mon[40587]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:26:45.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:44 smithi055 ceph-mon[40587]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:45 smithi067 ceph-mon[33346]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:45 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3099393225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:46.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:45 smithi055 ceph-mon[29406]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:46.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:45 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3099393225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:46.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:45 smithi055 ceph-mon[40587]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:46.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:45 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3099393225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:46 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:46 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]': finished 2024-03-26T21:26:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:46 smithi067 ceph-mon[33346]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:26:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:46 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:26:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:46 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/1452797009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]': finished 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[29406]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/1452797009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]: dispatch 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598"}]': finished 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[40587]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:26:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:46 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/1452797009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:26:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:47 smithi067 ceph-mon[33346]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:48.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:47 smithi055 ceph-mon[29406]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:48.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:47 smithi055 ceph-mon[40587]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:50.189 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:49 smithi067 ceph-mon[33346]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:50.283 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:49 smithi055 ceph-mon[29406]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:50.283 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:49 smithi055 ceph-mon[40587]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:51.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:51 smithi067 ceph-mon[33346]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:52.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:51 smithi055 ceph-mon[29406]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:51 smithi055 ceph-mon[40587]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:53.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:26:53.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:53.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:52 smithi067 ceph-mon[33346]: Deploying daemon osd.4 on smithi067 2024-03-26T21:26:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:26:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[29406]: Deploying daemon osd.4 on smithi067 2024-03-26T21:26:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T21:26:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:26:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:52 smithi055 ceph-mon[40587]: Deploying daemon osd.4 on smithi067 2024-03-26T21:26:54.178 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:53 smithi067 ceph-mon[33346]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:54.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:53 smithi055 ceph-mon[29406]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:53 smithi055 ceph-mon[40587]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:55 smithi067 ceph-mon[33346]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:56.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:55 smithi055 ceph-mon[29406]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:55 smithi055 ceph-mon[40587]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:58 smithi067 ceph-mon[33346]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[29406]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:26:58.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:26:58.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:58 smithi055 ceph-mon[40587]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:26:59 smithi067 ceph-mon[33346]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:00.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:26:59 smithi055 ceph-mon[40587]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:00.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:26:59 smithi055 ceph-mon[29406]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:01.064 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 4 on host 'smithi067' 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:27:01.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:01.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:01.675 DEBUG:teuthology.orchestra.run.smithi067:osd.4> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T21:27:01.677 INFO:tasks.cephadm:Deploying osd.5 on smithi067 with /dev/vg_nvme/lv_3... 2024-03-26T21:27:01.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-mon[33346]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:02.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[38347]: 2024-03-26T21:27:02.191+0000 7f66ec124700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T21:27:02.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:27:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[38347]: 2024-03-26T21:27:02.195+0000 7f66e7f17700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:27:02.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[29406]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: from='osd.4 v2:172.21.15.67:6800/1777724250' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:02.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:02.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:02 smithi055 ceph-mon[40587]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:02.799 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:03.404 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:03 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:03.404 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:03 smithi067 ceph-mon[33346]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T21:27:03.404 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:03 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.404 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:03 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.404 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:03 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[29406]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[40587]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:03 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: osd.4 v2:172.21.15.67:6800/1777724250 boot 2024-03-26T21:27:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:27:04.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:04.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:04 smithi067 ceph-mon[33346]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:04.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:04.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:04.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: osd.4 v2:172.21.15.67:6800/1777724250 boot 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[29406]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: osd.4 v2:172.21.15.67:6800/1777724250 boot 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:27:04.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:04 smithi055 ceph-mon[40587]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T21:27:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:05.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:05 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:27:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:05 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114713 s, 914 MB/s 2024-03-26T21:27:05.888 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-26T21:27:06.486 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:06 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi067 2024-03-26T21:27:06.486 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:06 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 17088M 2024-03-26T21:27:06.486 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:06 smithi067 ceph-mon[33346]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:27:06.486 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:06 smithi067 ceph-mon[33346]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:06.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-03-26T21:27:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi067 2024-03-26T21:27:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 17088M 2024-03-26T21:27:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[29406]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:27:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[29406]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi067 2024-03-26T21:27:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 17088M 2024-03-26T21:27:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[40587]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T21:27:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:06 smithi055 ceph-mon[40587]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:06.757 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:07 smithi067 ceph-mon[33346]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:27:07.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:07 smithi055 ceph-mon[29406]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:27:07.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:07 smithi055 ceph-mon[40587]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T21:27:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:08 smithi067 ceph-mon[33346]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T21:27:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:08 smithi067 ceph-mon[33346]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:08 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:08 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:08.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[29406]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[29406]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[40587]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[40587]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:08 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:09.457 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:09 smithi067 ceph-mon[33346]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T21:27:09.457 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:09 smithi067 ceph-mon[33346]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:09.457 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:09 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:09.457 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:09 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:09.457 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:09 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:09.750 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[29406]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[29406]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[40587]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[40587]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:09.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:09 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:10.718 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:10 smithi055 ceph-mon[29406]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-26T21:27:10.718 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:10 smithi055 ceph-mon[40587]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-26T21:27:10.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:10 smithi067 ceph-mon[33346]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-26T21:27:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:11 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/2949791257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]': finished 2024-03-26T21:27:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:11 smithi067 ceph-mon[33346]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:27:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/2949791257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]': finished 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[29406]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/2949791257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]: dispatch 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213"}]': finished 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[40587]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T21:27:11.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/2527701843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:12 smithi067 ceph-mon[33346]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T21:27:12.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/2527701843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:12.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:12 smithi055 ceph-mon[29406]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T21:27:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/2527701843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:12 smithi055 ceph-mon[40587]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T21:27:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:13 smithi067 ceph-mon[33346]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T21:27:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:13 smithi055 ceph-mon[40587]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T21:27:14.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:13 smithi055 ceph-mon[29406]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T21:27:16.478 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:16 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:16.478 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:16 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:16.478 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:16 smithi067 ceph-mon[33346]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T21:27:16.478 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:16.478 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:16 smithi067 ceph-mon[33346]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[40587]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[40587]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[29406]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:16 smithi055 ceph-mon[29406]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:17 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:27:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:17 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:18.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:17 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:27:18.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:17 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:18.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:17 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T21:27:18.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:17 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:18.906 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:18 smithi067 ceph-mon[33346]: Deploying daemon osd.5 on smithi067 2024-03-26T21:27:18.906 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:18 smithi067 ceph-mon[33346]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:19.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:18 smithi055 ceph-mon[29406]: Deploying daemon osd.5 on smithi067 2024-03-26T21:27:19.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:18 smithi055 ceph-mon[29406]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:19.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:18 smithi055 ceph-mon[40587]: Deploying daemon osd.5 on smithi067 2024-03-26T21:27:19.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:18 smithi055 ceph-mon[40587]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:19 smithi067 ceph-mon[33346]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:20.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:19 smithi055 ceph-mon[40587]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:20.203 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:19 smithi055 ceph-mon[29406]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:22.563 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:22 smithi067 ceph-mon[33346]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:22.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:22 smithi055 ceph-mon[29406]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:22.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:22 smithi055 ceph-mon[40587]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:24 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:24 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:24 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:24 smithi067 ceph-mon[33346]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[29406]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:24.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:24.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:24 smithi055 ceph-mon[40587]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:25 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:27:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:25 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:27:25.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:25 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:27:25.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:25 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:27:25.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:25 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:27:25.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:25 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:27:26.231 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:26 smithi067 ceph-mon[33346]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:26.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:26 smithi055 ceph-mon[40587]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:26.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:26 smithi055 ceph-mon[29406]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:26.621 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 5 on host 'smithi067' 2024-03-26T21:27:27.286 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.286 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.286 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.286 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.286 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.287 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:27 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.312 DEBUG:teuthology.orchestra.run.smithi067:osd.5> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T21:27:27.314 INFO:tasks.cephadm:Deploying osd.6 on smithi067 with /dev/vg_nvme/lv_2... 2024-03-26T21:27:27.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T21:27:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:27 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:28.080 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:27:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T21:27:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-mon[33346]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:28.490 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:27:28.195+0000 7fcfa3455700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T21:27:28.490 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:27:28 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:27:28.202+0000 7fcf9f248700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:27:28.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:27:28.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[29406]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: from='osd.5 v2:172.21.15.67:6804/278711358' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:28 smithi055 ceph-mon[40587]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T21:27:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T21:27:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: osd.5 v2:172.21.15.67:6804/278711358 boot 2024-03-26T21:27:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:27:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:29 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:29.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:29.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: osd.5 v2:172.21.15.67:6804/278711358 boot 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: osd.5 v2:172.21.15.67:6804/278711358 boot 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T21:27:29.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:29 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:27:30.430 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:30.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:30.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:30 smithi067 ceph-mon[33346]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:30.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[29406]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:30 smithi055 ceph-mon[40587]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:30.621 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T21:27:30.621 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T21:27:30.621 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-26T21:27:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-26T21:27:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0187134 s, 560 MB/s 2024-03-26T21:27:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-26T21:27:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:31 smithi067 ceph-mon[33346]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:27:31.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-03-26T21:27:31.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:31 smithi055 ceph-mon[29406]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:27:31.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:31 smithi055 ceph-mon[40587]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T21:27:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:32 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:32.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:32.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:32 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:33.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi067 2024-03-26T21:27:33.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 8544M 2024-03-26T21:27:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[29406]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T21:27:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi067 2024-03-26T21:27:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 8544M 2024-03-26T21:27:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:33 smithi055 ceph-mon[40587]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T21:27:33.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:33 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi067 2024-03-26T21:27:33.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:33 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 8544M 2024-03-26T21:27:33.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:33 smithi067 ceph-mon[33346]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T21:27:34.531 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:34.531 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:34.531 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:34.531 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:34.532 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:34.532 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:34 smithi067 ceph-mon[33346]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T21:27:34.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[29406]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:34.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:34 smithi055 ceph-mon[40587]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T21:27:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:35 smithi067 ceph-mon[33346]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:36.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:35 smithi055 ceph-mon[29406]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:35 smithi055 ceph-mon[40587]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3762227355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]': finished 2024-03-26T21:27:37.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T21:27:37.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:37.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3041231358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3762227355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]': finished 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3041231358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3762227355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]: dispatch 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9"}]': finished 2024-03-26T21:27:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T21:27:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3041231358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:27:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:38 smithi067 ceph-mon[33346]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:38 smithi055 ceph-mon[29406]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:38 smithi055 ceph-mon[40587]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:39.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:39 smithi067 ceph-mon[33346]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:40.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:39 smithi055 ceph-mon[29406]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:39 smithi055 ceph-mon[40587]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:41 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:41 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:41.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:41 smithi067 ceph-mon[33346]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T21:27:41.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:41 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:41.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:41 smithi067 ceph-mon[33346]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[40587]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[40587]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[29406]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T21:27:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:42.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:41 smithi055 ceph-mon[29406]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:43 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:27:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:43 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:43.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:43 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:27:43.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:43 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:43 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T21:27:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:43 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:44.758 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:44 smithi067 ceph-mon[33346]: Deploying daemon osd.6 on smithi067 2024-03-26T21:27:44.758 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:44 smithi067 ceph-mon[33346]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:44.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:44 smithi055 ceph-mon[40587]: Deploying daemon osd.6 on smithi067 2024-03-26T21:27:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:44 smithi055 ceph-mon[40587]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:44.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:44 smithi055 ceph-mon[29406]: Deploying daemon osd.6 on smithi067 2024-03-26T21:27:44.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:44 smithi055 ceph-mon[29406]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:46.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:45 smithi067 ceph-mon[33346]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:46.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:45 smithi055 ceph-mon[29406]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:46.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:45 smithi055 ceph-mon[40587]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:48.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:47 smithi067 ceph-mon[33346]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:47 smithi055 ceph-mon[40587]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:48.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:47 smithi055 ceph-mon[29406]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:49 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:49 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:49 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:49 smithi067 ceph-mon[33346]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[29406]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:50.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:49 smithi055 ceph-mon[40587]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:51 smithi067 ceph-mon[33346]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:52.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:51 smithi055 ceph-mon[29406]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:51 smithi055 ceph-mon[40587]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:52.830 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 6 on host 'smithi067' 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.138 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:52 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T21:27:53.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:52 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:53.464 DEBUG:teuthology.orchestra.run.smithi067:osd.6> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T21:27:53.466 INFO:tasks.cephadm:Deploying osd.7 on smithi067 with /dev/vg_nvme/lv_1... 2024-03-26T21:27:53.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T21:27:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:54.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:27:54.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T21:27:54.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:54.185 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.185 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:54.185 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-mon[33346]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:54.185 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:27:53.979+0000 7fa435513700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T21:27:54.185 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:27:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:27:53.985+0000 7fa431306700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[29406]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: from='osd.6 v2:172.21.15.67:6808/3926251639' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:54.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:53 smithi055 ceph-mon[40587]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:27:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T21:27:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: osd.6 v2:172.21.15.67:6808/3926251639 boot 2024-03-26T21:27:55.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:27:55.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:54 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: osd.6 v2:172.21.15.67:6808/3926251639 boot 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T21:27:55.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:55.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: osd.6 v2:172.21.15.67:6808/3926251639 boot 2024-03-26T21:27:55.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T21:27:55.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:54 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:27:56.171 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:27:56.171 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:27:56.171 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:56 smithi067 ceph-mon[33346]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:56.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:27:56.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:27:56.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[29406]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:27:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:27:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:55 smithi055 ceph-mon[40587]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123064 s, 852 MB/s 2024-03-26T21:27:56.837 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-03-26T21:27:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:57 smithi055 ceph-mon[40587]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T21:27:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:57 smithi055 ceph-mon[40587]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T21:27:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:57 smithi055 ceph-mon[29406]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T21:27:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:57 smithi055 ceph-mon[29406]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T21:27:57.459 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:57 smithi067 ceph-mon[33346]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T21:27:57.459 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:57 smithi067 ceph-mon[33346]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T21:27:57.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-03-26T21:27:58.124 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:58 smithi067 ceph-mon[33346]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:27:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:27:58.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:58 smithi055 ceph-mon[29406]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:27:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:58 smithi055 ceph-mon[40587]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:27:59.305 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T21:27:59.305 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi067 2024-03-26T21:27:59.305 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 5696M 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.306 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:27:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi067 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 5696M 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi067 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 5696M 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:27:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:27:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:27:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:27:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:27:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:00.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:00 smithi055 ceph-mon[29406]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T21:28:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:00 smithi055 ceph-mon[29406]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:28:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:00 smithi055 ceph-mon[40587]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T21:28:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:00 smithi055 ceph-mon[40587]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:28:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:00 smithi067 ceph-mon[33346]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T21:28:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:00 smithi067 ceph-mon[33346]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T21:28:01.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[29406]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[40587]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:28:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:01 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:01 smithi067 ceph-mon[33346]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T21:28:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T21:28:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:01 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:02 smithi067 ceph-mon[33346]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T21:28:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:02 smithi055 ceph-mon[29406]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T21:28:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:02 smithi055 ceph-mon[40587]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-26T21:28:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:03 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3668835803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:03 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:03 smithi067 ceph-mon[33346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]': finished 2024-03-26T21:28:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:03 smithi067 ceph-mon[33346]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T21:28:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:03 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3668835803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[29406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]': finished 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[29406]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3668835803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]: dispatch 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[40587]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55"}]': finished 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[40587]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T21:28:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:03 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:04.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3124329593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:28:04.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:04 smithi055 ceph-mon[29406]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:04.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3124329593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:28:04.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:04 smithi055 ceph-mon[40587]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3124329593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T21:28:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:04 smithi067 ceph-mon[33346]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:06.173 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:05 smithi067 ceph-mon[33346]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:05 smithi055 ceph-mon[29406]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:05 smithi055 ceph-mon[40587]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:07 smithi067 ceph-mon[33346]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:08.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:07 smithi055 ceph-mon[29406]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:08.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:07 smithi055 ceph-mon[40587]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:09.830 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:09 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:28:09.830 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:09 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:09 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:28:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:09 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:09 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T21:28:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:09 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:10.823 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:10 smithi067 ceph-mon[33346]: Deploying daemon osd.7 on smithi067 2024-03-26T21:28:10.823 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:10 smithi067 ceph-mon[33346]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:11.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:10 smithi055 ceph-mon[40587]: Deploying daemon osd.7 on smithi067 2024-03-26T21:28:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:10 smithi055 ceph-mon[40587]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:10 smithi055 ceph-mon[29406]: Deploying daemon osd.7 on smithi067 2024-03-26T21:28:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:10 smithi055 ceph-mon[29406]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:11 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:28:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:11 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:28:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:11 smithi067 ceph-mon[33346]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T21:28:11.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:11 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:12.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[29406]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[40587]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T21:28:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:11 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:12 smithi067 ceph-mon[33346]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:13.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:12 smithi055 ceph-mon[29406]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:12 smithi055 ceph-mon[40587]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:13 smithi067 ceph-mon[33346]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:14.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:13 smithi055 ceph-mon[29406]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:13 smithi055 ceph-mon[40587]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:16 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:16 smithi067 ceph-mon[33346]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:16.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:16.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[29406]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:16 smithi055 ceph-mon[40587]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:18.209 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:17 smithi067 ceph-mon[33346]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:18.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:17 smithi055 ceph-mon[29406]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:17 smithi055 ceph-mon[40587]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:18.935 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 7 on host 'smithi067' 2024-03-26T21:28:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:19 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:19.565 DEBUG:teuthology.orchestra.run.smithi067:osd.7> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T21:28:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:19.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:19 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:19.569 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T21:28:19.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T21:28:19.823 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:28:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:28:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[29406]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:20 smithi055 ceph-mon[40587]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:20.455 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:28:20.455 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: from='osd.7 v2:172.21.15.67:6812/2981645260' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.455 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T21:28:20.456 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:20.456 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:28:20.456 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-mon[33346]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T21:28:20.456 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[55899]: 2024-03-26T21:28:20.186+0000 7f477cc35700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T21:28:20.456 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:28:20 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[55899]: 2024-03-26T21:28:20.194+0000 7f4778a28700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:28:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:28:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: osd.7 v2:172.21.15.67:6812/2981645260 boot 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: osd.7 v2:172.21.15.67:6812/2981645260 boot 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:28:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:21 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: osd.7 v2:172.21.15.67:6812/2981645260 boot 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T21:28:21.342 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:21 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:28:21.914 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:28:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:22 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:28:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:22 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:28:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:22 smithi067 ceph-mon[33346]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:28:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:22 smithi067 ceph-mon[33346]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1896478908' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:28:22.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[29406]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[29406]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1896478908' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[40587]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[40587]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1896478908' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T21:28:22.646 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1711488501,"num_in_osds":8,"osd_in_since":1711488482,"num_remapped_pgs":1} 2024-03-26T21:28:22.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:28:22.908 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:28:23.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:28:23.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: Detected new or changed devices on smithi067 2024-03-26T21:28:23.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: Detected new or changed devices on smithi067 2024-03-26T21:28:23.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:23.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:23 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T21:28:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: Detected new or changed devices on smithi067 2024-03-26T21:28:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 4272M 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:23.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:23 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:24.818 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:28:24.818 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":63,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","created":"2024-03-26T21:22:56.673257+0000","modified":"2024-03-26T21:28:24.697392+0000","last_up_change":"2024-03-26T21:28:21.181246+0000","last_in_change":"2024-03-26T21:28:02.902888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:26:13.884991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f487071-c151-406b-b25b-3135cb3c63ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":317381212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":317381212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":317381212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":317381212}]},"public_addr":"172.21.15.55:6801/317381212","cluster_addr":"172.21.15.55:6802/317381212","heartbeat_back_addr":"172.21.15.55:6804/317381212","heartbeat_front_addr":"172.21.15.55:6803/317381212","state":["exists","up"]},{"osd":1,"uuid":"4fbc90fe-926a-4ac6-b5f0-d483ba4c09db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":3733554168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3733554168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3733554168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":3733554168}]},"public_addr":"172.21.15.55:6805/3733554168","cluster_addr":"172.21.15.55:6806/3733554168","heartbeat_back_addr":"172.21.15.55:6808/3733554168","heartbeat_front_addr":"172.21.15.55:6807/3733554168","state":["exists","up"]},{"osd":2,"uuid":"e2fe8018-1060-499c-a7e6-56d4da0eba7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6809","nonce":3059242923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3059242923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3059242923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6811","nonce":3059242923}]},"public_addr":"172.21.15.55:6809/3059242923","cluster_addr":"172.21.15.55:6810/3059242923","heartbeat_back_addr":"172.21.15.55:6812/3059242923","heartbeat_front_addr":"172.21.15.55:6811/3059242923","state":["exists","up"]},{"osd":3,"uuid":"30637a8d-5c1b-4ede-8214-a4595c875d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6813","nonce":3851321468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3851321468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3851321468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6815","nonce":3851321468}]},"public_addr":"172.21.15.55:6813/3851321468","cluster_addr":"172.21.15.55:6814/3851321468","heartbeat_back_addr":"172.21.15.55:6816/3851321468","heartbeat_front_addr":"172.21.15.55:6815/3851321468","state":["exists","up"]},{"osd":4,"uuid":"cd2d68be-4b10-4242-826b-e085c2ce5598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1777724250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6801","nonce":1777724250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":1777724250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1777724250}]},"public_addr":"172.21.15.67:6800/1777724250","cluster_addr":"172.21.15.67:6801/1777724250","heartbeat_back_addr":"172.21.15.67:6803/1777724250","heartbeat_front_addr":"172.21.15.67:6802/1777724250","state":["exists","up"]},{"osd":5,"uuid":"7468b94a-3216-4bc2-9b57-1d93e7c0b213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":278711358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":278711358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":278711358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":278711358}]},"public_addr":"172.21.15.67:6804/278711358","cluster_addr":"172.21.15.67:6805/278711358","heartbeat_back_addr":"172.21.15.67:6807/278711358","heartbeat_front_addr":"172.21.15.67:6806/278711358","state":["exists","up"]},{"osd":6,"uuid":"d47c79f3-96bf-46b6-a7f3-a315446d04b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3926251639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":3926251639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3926251639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3926251639}]},"public_addr":"172.21.15.67:6808/3926251639","cluster_addr":"172.21.15.67:6809/3926251639","heartbeat_back_addr":"172.21.15.67:6811/3926251639","heartbeat_front_addr":"172.21.15.67:6810/3926251639","state":["exists","up"]},{"osd":7,"uuid":"f10b98b8-887d-4f83-8bc9-99f5c302dc55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2981645260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2981645260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2981645260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2981645260}]},"public_addr":"172.21.15.67:6812/2981645260","cluster_addr":"172.21.15.67:6813/2981645260","heartbeat_back_addr":"172.21.15.67:6815/2981645260","heartbeat_front_addr":"172.21.15.67:6814/2981645260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:40.766230+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:08.184580+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:15.138703+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:51.968364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:10.190062+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:27.265244+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:53.118680+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:19.212377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6800/2804789175":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/2199694469":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/1614234257":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/2834138692":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/1740219825":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/268664696":"2024-03-27T21:23:57.787077+0000","172.21.15.55:6800/536717527":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/559658018":"2024-03-27T21:23:57.787077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:28:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:24 smithi067 ceph-mon[33346]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:28:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:24 smithi067 ceph-mon[33346]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:25.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:24 smithi055 ceph-mon[29406]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:28:25.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:24 smithi055 ceph-mon[29406]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:24 smithi055 ceph-mon[40587]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T21:28:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:24 smithi055 ceph-mon[40587]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:25.590 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T21:28:25.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T21:28:25.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:25 smithi055 ceph-mon[40587]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:28:25.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:25 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2913562020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:28:25.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:25 smithi055 ceph-mon[29406]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:28:25.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:25 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2913562020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:28:25.842 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:28:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:25 smithi067 ceph-mon[33346]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T21:28:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:25 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2913562020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:28:26.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[29406]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:27.000 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:28:27.000 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:28:27.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[40587]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:27.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:28:27.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:26 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:28:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:26 smithi067 ceph-mon[33346]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:26 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:28:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:26 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:28:27.634 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2024-03-26T21:28:28.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[40587]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3745145100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[29406]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:28:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3745145100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:28:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:27 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:28:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:27 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:28:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:27 smithi067 ceph-mon[33346]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T21:28:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3745145100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T21:28:28.340 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi055 2024-03-26T21:28:28.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi055=foo.a' 2024-03-26T21:28:28.596 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:28 smithi067 ceph-mon[33346]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T21:28:29.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:28 smithi055 ceph-mon[29406]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T21:28:29.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:28 smithi055 ceph-mon[40587]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T21:28:29.964 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:29 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:28:29.964 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:29 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:28:30.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:29 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:28:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:29 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:28:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:29 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:28:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:29 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:28:30.392 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled rgw.foo.a update... 2024-03-26T21:28:31.058 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:30 smithi067 ceph-mon[33346]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:28:31.059 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:30 smithi067 ceph-mon[33346]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:28:31.059 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:30 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.059 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:30 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[29406]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:28:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[29406]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:28:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[40587]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T21:28:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[40587]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T21:28:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:30 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:31.085 DEBUG:teuthology.orchestra.run.smithi055:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T21:28:31.087 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi067 2024-03-26T21:28:31.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T21:28:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:31.958 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi055=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:31.958 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:31.958 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi055=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:28:31.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:28:31.960 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.960 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:31 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi055=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:31 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:32.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:32 smithi055 ceph-mon[29406]: Deploying daemon rgw.foo.a on smithi055 2024-03-26T21:28:32.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:32 smithi055 ceph-mon[29406]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-26T21:28:32.874 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:32 smithi055 ceph-mon[40587]: Deploying daemon rgw.foo.a on smithi055 2024-03-26T21:28:32.874 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:32 smithi055 ceph-mon[40587]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-26T21:28:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:32 smithi067 ceph-mon[33346]: Deploying daemon rgw.foo.a on smithi055 2024-03-26T21:28:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:32 smithi067 ceph-mon[33346]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-26T21:28:33.308 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 systemd[1]: Starting Ceph rgw.foo.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:28:33.565 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 podman[67262]: 2024-03-26 21:28:33.315859491 +0000 UTC m=+0.107336969 container create 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T21:28:33.566 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 podman[67262]: 2024-03-26 21:28:33.226539428 +0000 UTC m=+0.018016864 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:28:33.566 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 podman[67262]: 2024-03-26 21:28:33.551701416 +0000 UTC m=+0.343178833 container init 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T21:28:33.566 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 podman[67262]: 2024-03-26 21:28:33.55428137 +0000 UTC m=+0.345758786 container start 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:28:33.823 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'datapool' created 2024-03-26T21:28:33.974 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 bash[67262]: 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 2024-03-26T21:28:33.975 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:28:33 smithi055 systemd[1]: Started Ceph rgw.foo.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/3288271008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.975 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/3288271008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:33.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:33 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/3288271008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.123 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:33 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:34.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T21:28:34.686 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:35.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[40587]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:28:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:35.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:34 smithi055 ceph-mon[29406]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: Saving service rgw.foo.a spec with placement smithi055=foo.a;count:1 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:35.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:34 smithi067 ceph-mon[33346]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:36.174 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:36.175 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:28:36.175 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.175 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.175 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:35 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1536543168' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:36.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:35 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: Checking dashboard <-> RGW credentials 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: pgmap v197: 68 pgs: 11 creating+peering, 7 active+clean, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/1278641844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:28:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:36 smithi067 ceph-mon[33346]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:28:37.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: Checking dashboard <-> RGW credentials 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: pgmap v197: 68 pgs: 11 creating+peering, 7 active+clean, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/1278641844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[29406]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: Checking dashboard <-> RGW credentials 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: pgmap v197: 68 pgs: 11 creating+peering, 7 active+clean, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/1278641844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T21:28:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T21:28:37.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T21:28:37.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:36 smithi055 ceph-mon[40587]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T21:28:39.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:28:39.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: pgmap v200: 100 pgs: 14 creating+peering, 27 active+clean, 59 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: pgmap v200: 100 pgs: 14 creating+peering, 27 active+clean, 59 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:28:39.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:38 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T21:28:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T21:28:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: pgmap v200: 100 pgs: 14 creating+peering, 27 active+clean, 59 unknown; 578 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T21:28:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:38 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:39.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.67 --placement '1;smithi067=iscsi.a' 2024-03-26T21:28:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:39.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:39 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:39.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:39 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:39.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:39 smithi067 ceph-mon[33346]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:28:40.129 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:40.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:40.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[40587]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:28:40.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:40.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T21:28:40.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:39 smithi055 ceph-mon[29406]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T21:28:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: pgmap v202: 100 pgs: 7 creating+peering, 72 active+clean, 21 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T21:28:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:28:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:40.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:28:40.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:40 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:28:41.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: pgmap v202: 100 pgs: 7 creating+peering, 72 active+clean, 21 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: pgmap v202: 100 pgs: 7 creating+peering, 72 active+clean, 21 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T21:28:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:28:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:40 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:28:41.952 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled iscsi.datapool update... 2024-03-26T21:28:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:28:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:41 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3752573601' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:41 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T21:28:42.673 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T21:28:42.673 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:28:42.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T21:28:42.706 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:28:42.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T21:28:42.738 DEBUG:teuthology.orchestra.run.smithi067:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T21:28:42.780 INFO:tasks.cephadm:Adding prometheus.a on smithi067 2024-03-26T21:28:42.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi067=a' 2024-03-26T21:28:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: pgmap v205: 132 pgs: 11 creating+peering, 99 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 2 op/s 2024-03-26T21:28:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.67", "placement": "1;smithi067=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: Saving service iscsi.datapool spec with placement smithi067=iscsi.a;count:1 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:42 smithi067 ceph-mon[33346]: Deploying daemon iscsi.iscsi.a on smithi067 2024-03-26T21:28:43.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: pgmap v205: 132 pgs: 11 creating+peering, 99 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 2 op/s 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.67", "placement": "1;smithi067=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: Saving service iscsi.datapool spec with placement smithi067=iscsi.a;count:1 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:28:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[29406]: Deploying daemon iscsi.iscsi.a on smithi067 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: pgmap v205: 132 pgs: 11 creating+peering, 99 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 2 op/s 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1460561447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.67", "placement": "1;smithi067=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: Saving service iscsi.datapool spec with placement smithi067=iscsi.a;count:1 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:43.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T21:28:43.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T21:28:43.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:43.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:42 smithi055 ceph-mon[40587]: Deploying daemon iscsi.iscsi.a on smithi067 2024-03-26T21:28:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:44.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:43 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:28:44.185 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:43 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:28:44.185 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:43 smithi067 ceph-mon[33346]: pgmap v207: 132 pgs: 7 creating+peering, 114 active+clean, 11 unknown; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.6 KiB/s wr, 78 op/s 2024-03-26T21:28:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:28:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:28:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[29406]: pgmap v207: 132 pgs: 7 creating+peering, 114 active+clean, 11 unknown; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.6 KiB/s wr, 78 op/s 2024-03-26T21:28:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:28:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:28:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:43 smithi055 ceph-mon[40587]: pgmap v207: 132 pgs: 7 creating+peering, 114 active+clean, 11 unknown; 580 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.6 KiB/s wr, 78 op/s 2024-03-26T21:28:44.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:44 smithi067 systemd[1]: Starting Ceph iscsi.iscsi.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:28:45.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:45 smithi067 podman[60639]: 2024-03-26 21:28:45.490853475 +0000 UTC m=+0.021883004 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:28:45.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:45 smithi067 podman[60639]: 2024-03-26 21:28:45.594416174 +0000 UTC m=+0.125445690 container create 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T21:28:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:45 smithi067 ceph-mon[33346]: pgmap v208: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.5 KiB/s wr, 166 op/s 2024-03-26T21:28:46.194 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled prometheus update... 2024-03-26T21:28:46.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:45 smithi055 ceph-mon[29406]: pgmap v208: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.5 KiB/s wr, 166 op/s 2024-03-26T21:28:46.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:45 smithi055 ceph-mon[40587]: pgmap v208: 132 pgs: 132 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.5 KiB/s wr, 166 op/s 2024-03-26T21:28:46.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 podman[60639]: 2024-03-26 21:28:46.074139391 +0000 UTC m=+0.605168910 container init 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:28:46.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 podman[60639]: 2024-03-26 21:28:46.077316161 +0000 UTC m=+0.608345680 container start 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:28:46.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 bash[60639]: 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 2024-03-26T21:28:46.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 systemd[1]: Started Ceph iscsi.iscsi.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:28:46.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Started the configuration object watcher 2024-03-26T21:28:46.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Checking for config object changes every 1s 2024-03-26T21:28:46.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Processing osd blocklist entries for this node 2024-03-26T21:28:46.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Reading the configuration object to update local LIO configuration 2024-03-26T21:28:46.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Configuration does not have an entry for this host(smithi067.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T21:28:46.906 DEBUG:teuthology.orchestra.run.smithi067:prometheus.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T21:28:46.908 INFO:tasks.cephadm:Adding node-exporter.a on smithi055 2024-03-26T21:28:46.908 INFO:tasks.cephadm:Adding node-exporter.b on smithi067 2024-03-26T21:28:46.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi055=a;smithi067=b' 2024-03-26T21:28:46.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T21:28:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: Saving service prometheus spec with placement smithi067=a;count:1 2024-03-26T21:28:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:28:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:47 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/1172421134' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: Saving service prometheus spec with placement smithi067=a;count:1 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/1172421134' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: Saving service prometheus spec with placement smithi067=a;count:1 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:28:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T21:28:47.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:47 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/1172421134' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T21:28:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:48 smithi067 ceph-mon[33346]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.2 KiB/s wr, 161 op/s 2024-03-26T21:28:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:48 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:48 smithi055 ceph-mon[29406]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.2 KiB/s wr, 161 op/s 2024-03-26T21:28:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:48 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:48 smithi055 ceph-mon[40587]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.2 KiB/s wr, 161 op/s 2024-03-26T21:28:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:48 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:49 smithi067 ceph-mon[33346]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:28:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:49 smithi055 ceph-mon[29406]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:28:49.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:49 smithi055 ceph-mon[40587]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T21:28:49.994 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled node-exporter update... 2024-03-26T21:28:50.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:50 smithi055 ceph-mon[29406]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-26T21:28:50.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:50 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:50.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:50 smithi055 ceph-mon[40587]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-26T21:28:50.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:50 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:50 smithi067 ceph-mon[33346]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-26T21:28:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:50 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:51.229 DEBUG:teuthology.orchestra.run.smithi055:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T21:28:51.231 DEBUG:teuthology.orchestra.run.smithi067:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T21:28:51.271 INFO:tasks.cephadm:Adding alertmanager.a on smithi055 2024-03-26T21:28:51.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi055=a' 2024-03-26T21:28:51.568 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:51.979 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi055=a;smithi067=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: Saving service node-exporter spec with placement smithi055=a;smithi067=b;count:2 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:51.980 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:51 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi055=a;smithi067=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: Saving service node-exporter spec with placement smithi055=a;smithi067=b;count:2 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi055=a;smithi067=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: Saving service node-exporter spec with placement smithi055=a;smithi067=b;count:2 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:52.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:28:52.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:28:52.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:51 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:53.045 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:52 smithi067 ceph-mon[33346]: Deploying daemon prometheus.a on smithi067 2024-03-26T21:28:53.045 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:52 smithi067 ceph-mon[33346]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-03-26T21:28:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:52 smithi055 ceph-mon[29406]: Deploying daemon prometheus.a on smithi067 2024-03-26T21:28:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:52 smithi055 ceph-mon[29406]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-03-26T21:28:53.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:52 smithi055 ceph-mon[40587]: Deploying daemon prometheus.a on smithi067 2024-03-26T21:28:53.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:52 smithi055 ceph-mon[40587]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-03-26T21:28:53.735 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled alertmanager update... 2024-03-26T21:28:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:53 smithi067 ceph-mon[33346]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:53 smithi067 ceph-mon[33346]: Saving service alertmanager spec with placement smithi055=a;count:1 2024-03-26T21:28:53.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:53 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:53.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:53 smithi067 ceph-mon[33346]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 2.1 KiB/s wr, 74 op/s 2024-03-26T21:28:54.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[40587]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[40587]: Saving service alertmanager spec with placement smithi055=a;count:1 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[40587]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 2.1 KiB/s wr, 74 op/s 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[29406]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi055=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[29406]: Saving service alertmanager spec with placement smithi055=a;count:1 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:53 smithi055 ceph-mon[29406]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 2.1 KiB/s wr, 74 op/s 2024-03-26T21:28:54.765 DEBUG:teuthology.orchestra.run.smithi055:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T21:28:54.768 INFO:tasks.cephadm:Adding grafana.a on smithi067 2024-03-26T21:28:54.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi067=a' 2024-03-26T21:28:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:28:56.203 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:55 smithi067 ceph-mon[33346]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 409 B/s wr, 13 op/s 2024-03-26T21:28:56.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:55 smithi055 ceph-mon[40587]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 409 B/s wr, 13 op/s 2024-03-26T21:28:56.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:55 smithi055 ceph-mon[29406]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 409 B/s wr, 13 op/s 2024-03-26T21:28:56.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:28:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:28:57.126 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:56 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:57.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:56 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:56 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:28:57.626 INFO:teuthology.orchestra.run.smithi067.stdout:Scheduled grafana update... 2024-03-26T21:28:58.300 DEBUG:teuthology.orchestra.run.smithi067:grafana.a> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T21:28:58.303 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T21:28:58.303 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T21:28:58.557 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:28:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: Saving service grafana spec with placement smithi067=a;count:1 2024-03-26T21:28:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-26T21:28:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:28:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T21:28:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:58 smithi067 ceph-mon[33346]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 89 B/s wr, 1 op/s 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: Saving service grafana spec with placement smithi067=a;count:1 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[40587]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 89 B/s wr, 1 op/s 2024-03-26T21:28:59.002 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi067=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: Saving service grafana spec with placement smithi067=a;count:1 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T21:28:59.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:58 smithi055 ceph-mon[29406]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 89 B/s wr, 1 op/s 2024-03-26T21:28:59.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-26T21:28:59.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:28:59.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:59 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T21:28:59.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:28:59 smithi067 ceph-mon[33346]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[40587]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-26T21:29:00.007 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T21:29:00.008 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:28:59 smithi055 ceph-mon[29406]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T21:29:00.667 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-03-26T21:29:00.667 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAcPgNmpoLvJxAA3u3diy7WNyjFNlDma7dCgg== 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[40587]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[40587]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3003405849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[29406]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:29:00.980 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[29406]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:00.981 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3003405849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.981 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.981 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:00 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:00 smithi067 ceph-mon[33346]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T21:29:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:00 smithi067 ceph-mon[33346]: pgmap v218: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3003405849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:00 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:00 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:01.340 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-26T21:29:01.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T21:29:01.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T21:29:01.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T21:29:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:02 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:02 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:02 smithi067 ceph-mon[33346]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:29:02.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[29406]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:29:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:02 smithi055 ceph-mon[40587]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:29:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:03 smithi067 ceph-mon[33346]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:29:03.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:03 smithi055 ceph-mon[40587]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:29:03.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:03 smithi055 ceph-mon[29406]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T21:29:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:29:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:04 smithi067 ceph-mon[33346]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T21:29:04.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:04 smithi055 ceph-mon[40587]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T21:29:04.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:04 smithi055 ceph-mon[29406]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T21:29:05.099 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:05 smithi067 systemd[1]: Starting Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:29:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:05 smithi067 ceph-mon[33346]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s; 49 B/s, 2 objects/s recovering 2024-03-26T21:29:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:05 smithi055 ceph-mon[40587]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s; 49 B/s, 2 objects/s recovering 2024-03-26T21:29:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:05 smithi055 ceph-mon[29406]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 711 B/s rd, 0 op/s; 49 B/s, 2 objects/s recovering 2024-03-26T21:29:06.357 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 podman[62261]: 2024-03-26 21:29:06.050687056 +0000 UTC m=+0.019711790 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T21:29:06.357 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 podman[62261]: 2024-03-26 21:29:06.365169092 +0000 UTC m=+0.334193824 container create 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:06.588 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-03-26T21:29:06.588 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAiPgNmDvG6IhAApa41iLlk2R6FjxThohLC+w== 2024-03-26T21:29:06.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 podman[62261]: 2024-03-26 21:29:06.625779839 +0000 UTC m=+0.594804573 container init 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 podman[62261]: 2024-03-26 21:29:06.628636118 +0000 UTC m=+0.597660859 container start 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.652Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.652Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.652Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi067 (none))" 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.652Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.652Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T21:29:06.878 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.653Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.653Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.654Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.654Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.657Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.657Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.103µs 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.657Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.657Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.657Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.132µs wal_replay_duration=211.796µs wbl_replay_duration=89ns total_replay_duration=248.545µs 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.658Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.658Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.658Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.677Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.627601ms db_storage=705ns remote_storage=1.052µs web_handler=452ns query_engine=536ns scrape=385.842µs scrape_sd=74.061µs notify=703ns notify_sd=1.446µs rules=18.919809ms tracing=5.872µs 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.677Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T21:29:06.879 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:29:06.677Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T21:29:06.880 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 bash[62261]: 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d 2024-03-26T21:29:07.239 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:29:06 smithi067 systemd[1]: Started Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:29:07.364 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-26T21:29:07.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T21:29:07.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T21:29:07.407 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T21:29:07.407 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T21:29:07.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T21:29:07.662 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:29:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.67:0/2382150307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:07 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:29:07.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.67:0/2382150307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.67:0/2382150307' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:07.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:07 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T21:29:08.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setuser ceph since I am not root 2024-03-26T21:29:08.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ignoring --setgroup ceph since I am not root 2024-03-26T21:29:08.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: ignoring --setuser ceph since I am not root 2024-03-26T21:29:08.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: ignoring --setgroup ceph since I am not root 2024-03-26T21:29:08.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-mgr[35199]: -- 172.21.15.67:0/1628578401 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55844f3345a0 con 0x55844f308800 2024-03-26T21:29:08.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[40587]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T21:29:08.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:08.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[40587]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:29:08.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[40587]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[29406]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[29406]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-mon[29406]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:08.586+0000 7ffa36b75200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:29:08.802 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:08.683+0000 7ffa36b75200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:29:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-mon[33346]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T21:29:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' 2024-03-26T21:29:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-mon[33346]: from='mgr.14152 172.21.15.55:0/116784611' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T21:29:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-mon[33346]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T21:29:08.990 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:08.586+0000 7f038d8ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T21:29:08.990 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:08.682+0000 7f038d8ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T21:29:09.487 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:29:09.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1027976489' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:29:09.676 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:09.344+0000 7f038d8ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:29:09.676 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:09.440+0000 7f038d8ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:29:09.676 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:09.585+0000 7f038d8ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:29:09.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1027976489' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:29:09.723 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1027976489' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T21:29:09.723 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:09.385+0000 7ffa36b75200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T21:29:09.723 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:09.480+0000 7ffa36b75200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T21:29:09.724 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:09.632+0000 7ffa36b75200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T21:29:09.724 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:09.729+0000 7ffa36b75200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:29:09.948 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:09.682+0000 7f038d8ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T21:29:09.948 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:09.772+0000 7f038d8ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:29:10.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:09.820+0000 7ffa36b75200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T21:29:10.239 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:10.050+0000 7f038d8ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:29:10.239 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:10.168+0000 7f038d8ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:29:10.290 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":147833426}]},"active_addr":"172.21.15.55:6800/147833426","active_change":"2024-03-26T21:23:57.787228+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.55:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2854772543}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":4141219490}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":966571165}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2757111431}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2649740776}]}]} 2024-03-26T21:29:10.295 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T21:29:10.295 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T21:29:10.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:29:10.394 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:10.092+0000 7ffa36b75200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T21:29:10.394 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:10.205+0000 7ffa36b75200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T21:29:10.395 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:10.295+0000 7ffa36b75200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:29:10.529 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:10.259+0000 7f038d8ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T21:29:10.529 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:10.360+0000 7f038d8ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:29:10.562 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:10.684 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:10.684 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:10.684 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[40587]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:29:10.685 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:10.685 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:10.685 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-mon[29406]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:29:10.685 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:10.400+0000 7ffa36b75200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T21:29:10.685 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:10.603+0000 7ffa36b75200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:29:10.932 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:10.932 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:10.932 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-mon[33346]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T21:29:10.933 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:10.541+0000 7f038d8ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T21:29:11.873 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:11 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:11.633+0000 7ffa36b75200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:29:11.873 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:11 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:11.774+0000 7ffa36b75200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:29:11.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:11.547+0000 7f038d8ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T21:29:11.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:11.680+0000 7f038d8ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T21:29:12.157 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:12.123+0000 7ffa36b75200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:29:12.308 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:12.021+0000 7f038d8ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T21:29:12.308 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:12.113+0000 7f038d8ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:29:12.308 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:12.314+0000 7f038d8ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:29:12.423 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:12.218+0000 7ffa36b75200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T21:29:12.424 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:12.430+0000 7ffa36b75200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:29:12.484 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:29:12.484 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":81,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","created":"2024-03-26T21:22:56.673257+0000","modified":"2024-03-26T21:29:11.479166+0000","last_up_change":"2024-03-26T21:28:21.181246+0000","last_in_change":"2024-03-26T21:28:02.902888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:26:13.884991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T21:28:33.142617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T21:28:33.591599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:28:34.904126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:28:36.852373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:28:38.937273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f487071-c151-406b-b25b-3135cb3c63ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":317381212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":317381212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":317381212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":317381212}]},"public_addr":"172.21.15.55:6801/317381212","cluster_addr":"172.21.15.55:6802/317381212","heartbeat_back_addr":"172.21.15.55:6804/317381212","heartbeat_front_addr":"172.21.15.55:6803/317381212","state":["exists","up"]},{"osd":1,"uuid":"4fbc90fe-926a-4ac6-b5f0-d483ba4c09db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":3733554168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3733554168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3733554168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":3733554168}]},"public_addr":"172.21.15.55:6805/3733554168","cluster_addr":"172.21.15.55:6806/3733554168","heartbeat_back_addr":"172.21.15.55:6808/3733554168","heartbeat_front_addr":"172.21.15.55:6807/3733554168","state":["exists","up"]},{"osd":2,"uuid":"e2fe8018-1060-499c-a7e6-56d4da0eba7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6809","nonce":3059242923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3059242923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3059242923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6811","nonce":3059242923}]},"public_addr":"172.21.15.55:6809/3059242923","cluster_addr":"172.21.15.55:6810/3059242923","heartbeat_back_addr":"172.21.15.55:6812/3059242923","heartbeat_front_addr":"172.21.15.55:6811/3059242923","state":["exists","up"]},{"osd":3,"uuid":"30637a8d-5c1b-4ede-8214-a4595c875d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6813","nonce":3851321468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3851321468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3851321468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6815","nonce":3851321468}]},"public_addr":"172.21.15.55:6813/3851321468","cluster_addr":"172.21.15.55:6814/3851321468","heartbeat_back_addr":"172.21.15.55:6816/3851321468","heartbeat_front_addr":"172.21.15.55:6815/3851321468","state":["exists","up"]},{"osd":4,"uuid":"cd2d68be-4b10-4242-826b-e085c2ce5598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1777724250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6801","nonce":1777724250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":1777724250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1777724250}]},"public_addr":"172.21.15.67:6800/1777724250","cluster_addr":"172.21.15.67:6801/1777724250","heartbeat_back_addr":"172.21.15.67:6803/1777724250","heartbeat_front_addr":"172.21.15.67:6802/1777724250","state":["exists","up"]},{"osd":5,"uuid":"7468b94a-3216-4bc2-9b57-1d93e7c0b213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":278711358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":278711358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":278711358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":278711358}]},"public_addr":"172.21.15.67:6804/278711358","cluster_addr":"172.21.15.67:6805/278711358","heartbeat_back_addr":"172.21.15.67:6807/278711358","heartbeat_front_addr":"172.21.15.67:6806/278711358","state":["exists","up"]},{"osd":6,"uuid":"d47c79f3-96bf-46b6-a7f3-a315446d04b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3926251639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":3926251639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3926251639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3926251639}]},"public_addr":"172.21.15.67:6808/3926251639","cluster_addr":"172.21.15.67:6809/3926251639","heartbeat_back_addr":"172.21.15.67:6811/3926251639","heartbeat_front_addr":"172.21.15.67:6810/3926251639","state":["exists","up"]},{"osd":7,"uuid":"f10b98b8-887d-4f83-8bc9-99f5c302dc55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2981645260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2981645260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2981645260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2981645260}]},"public_addr":"172.21.15.67:6812/2981645260","cluster_addr":"172.21.15.67:6813/2981645260","heartbeat_back_addr":"172.21.15.67:6815/2981645260","heartbeat_front_addr":"172.21.15.67:6814/2981645260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:29:01.676554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:26.055014+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:25.961079+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:29:09.052472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:10.190062+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:29.153623+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:53.118680+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:19.212377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:6800/2804789175":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/2199694469":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/1614234257":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/2834138692":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/1740219825":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/268664696":"2024-03-27T21:23:57.787077+0000","172.21.15.55:6800/536717527":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/559658018":"2024-03-27T21:23:57.787077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:29:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-mon[33346]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:29:12.739 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:12.505+0000 7f038d8ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:29:12.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-mon[29406]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:29:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-mon[40587]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T21:29:12.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:12.626+0000 7ffa36b75200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T21:29:13.054 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:12.796+0000 7f038d8ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:29:13.183 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:12.926+0000 7ffa36b75200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T21:29:13.267 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T21:29:13.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:29:13.437 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:13 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:13.189+0000 7ffa36b75200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:29:13.438 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:13 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:13.441+0000 7ffa36b75200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:29:13.479 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:13.060+0000 7f038d8ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T21:29:13.479 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:13.305+0000 7f038d8ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T21:29:13.520 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:13.734 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3259459126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:13.734 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:13 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3259459126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:13.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:13 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3259459126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:14.489 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:14.135+0000 7f038d8ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:29:14.565 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:14 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:14.302+0000 7ffa36b75200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T21:29:14.946 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:14 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:14.899+0000 7ffa36b75200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:29:14.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:14.718+0000 7f038d8ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T21:29:14.989 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:14.818+0000 7f038d8ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: Standby manager daemon x restarted 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: Standby manager daemon x started 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[40587]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:29:15.231 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: Standby manager daemon x restarted 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: Standby manager daemon x started 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-mon[29406]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:29:15.232 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:15.002+0000 7ffa36b75200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T21:29:15.376 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: Standby manager daemon x restarted 2024-03-26T21:29:15.376 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: Standby manager daemon x started 2024-03-26T21:29:15.376 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-mon[33346]: from='mgr.? 172.21.15.67:0/2305102924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:15.042+0000 7f038d8ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: [26/Mar/2024:21:29:15] ENGINE Bus STARTING 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: CherryPy Checker: 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: The Application mounted at '' has an empty config. 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: [26/Mar/2024:21:29:15] ENGINE Serving on http://:::9283 2024-03-26T21:29:15.377 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:29:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x[35195]: [26/Mar/2024:21:29:15] ENGINE Bus STARTED 2024-03-26T21:29:15.444 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:29:15.444 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":82,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","created":"2024-03-26T21:22:56.673257+0000","modified":"2024-03-26T21:29:15.251926+0000","last_up_change":"2024-03-26T21:28:21.181246+0000","last_in_change":"2024-03-26T21:28:02.902888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:26:13.884991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T21:28:33.142617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T21:28:33.591599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:28:34.904126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:28:36.852373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:28:38.937273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f487071-c151-406b-b25b-3135cb3c63ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":317381212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":317381212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":317381212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":317381212}]},"public_addr":"172.21.15.55:6801/317381212","cluster_addr":"172.21.15.55:6802/317381212","heartbeat_back_addr":"172.21.15.55:6804/317381212","heartbeat_front_addr":"172.21.15.55:6803/317381212","state":["exists","up"]},{"osd":1,"uuid":"4fbc90fe-926a-4ac6-b5f0-d483ba4c09db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":3733554168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3733554168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3733554168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":3733554168}]},"public_addr":"172.21.15.55:6805/3733554168","cluster_addr":"172.21.15.55:6806/3733554168","heartbeat_back_addr":"172.21.15.55:6808/3733554168","heartbeat_front_addr":"172.21.15.55:6807/3733554168","state":["exists","up"]},{"osd":2,"uuid":"e2fe8018-1060-499c-a7e6-56d4da0eba7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6809","nonce":3059242923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3059242923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3059242923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6811","nonce":3059242923}]},"public_addr":"172.21.15.55:6809/3059242923","cluster_addr":"172.21.15.55:6810/3059242923","heartbeat_back_addr":"172.21.15.55:6812/3059242923","heartbeat_front_addr":"172.21.15.55:6811/3059242923","state":["exists","up"]},{"osd":3,"uuid":"30637a8d-5c1b-4ede-8214-a4595c875d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6813","nonce":3851321468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3851321468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3851321468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6815","nonce":3851321468}]},"public_addr":"172.21.15.55:6813/3851321468","cluster_addr":"172.21.15.55:6814/3851321468","heartbeat_back_addr":"172.21.15.55:6816/3851321468","heartbeat_front_addr":"172.21.15.55:6815/3851321468","state":["exists","up"]},{"osd":4,"uuid":"cd2d68be-4b10-4242-826b-e085c2ce5598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1777724250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6801","nonce":1777724250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":1777724250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1777724250}]},"public_addr":"172.21.15.67:6800/1777724250","cluster_addr":"172.21.15.67:6801/1777724250","heartbeat_back_addr":"172.21.15.67:6803/1777724250","heartbeat_front_addr":"172.21.15.67:6802/1777724250","state":["exists","up"]},{"osd":5,"uuid":"7468b94a-3216-4bc2-9b57-1d93e7c0b213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":278711358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":278711358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":278711358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":278711358}]},"public_addr":"172.21.15.67:6804/278711358","cluster_addr":"172.21.15.67:6805/278711358","heartbeat_back_addr":"172.21.15.67:6807/278711358","heartbeat_front_addr":"172.21.15.67:6806/278711358","state":["exists","up"]},{"osd":6,"uuid":"d47c79f3-96bf-46b6-a7f3-a315446d04b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3926251639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":3926251639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3926251639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3926251639}]},"public_addr":"172.21.15.67:6808/3926251639","cluster_addr":"172.21.15.67:6809/3926251639","heartbeat_back_addr":"172.21.15.67:6811/3926251639","heartbeat_front_addr":"172.21.15.67:6810/3926251639","state":["exists","up"]},{"osd":7,"uuid":"f10b98b8-887d-4f83-8bc9-99f5c302dc55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2981645260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2981645260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2981645260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2981645260}]},"public_addr":"172.21.15.67:6812/2981645260","cluster_addr":"172.21.15.67:6813/2981645260","heartbeat_back_addr":"172.21.15.67:6815/2981645260","heartbeat_front_addr":"172.21.15.67:6814/2981645260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:29:01.676554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:26.055014+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:25.961079+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:29:09.052472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:10.190062+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:29.153623+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:27:53.118680+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:28:19.212377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2649740776":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/4141219490":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/147833426":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2854772543":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/2804789175":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/2199694469":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/1614234257":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/2834138692":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/1740219825":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/268664696":"2024-03-27T21:23:57.787077+0000","172.21.15.55:6800/536717527":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/966571165":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2757111431":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/559658018":"2024-03-27T21:23:57.787077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:29:15.481 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:15.237+0000 7ffa36b75200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T21:29:15.481 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:15] ENGINE Bus STARTING 2024-03-26T21:29:15.815 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: CherryPy Checker: 2024-03-26T21:29:15.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: The Application mounted at '' has an empty config. 2024-03-26T21:29:15.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:29:15.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:15] ENGINE Serving on http://:::9283 2024-03-26T21:29:15.816 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:15] ENGINE Bus STARTED 2024-03-26T21:29:16.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T21:29:16.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T21:29:16.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T21:29:16.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T21:29:16.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T21:29:16.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T21:29:16.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T21:29:16.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: Active manager daemon y restarted 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: Activating manager daemon y 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: mgrmap e19: y(active, starting, since 0.00976175s), standbys: x 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:29:16.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: Manager daemon y is now available 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.392 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.392 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3665716841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:29:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: Active manager daemon y restarted 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: Activating manager daemon y 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: mgrmap e19: y(active, starting, since 0.00976175s), standbys: x 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:29:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: Manager daemon y is now available 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3665716841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: Active manager daemon y restarted 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: Activating manager daemon y 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: mgrmap e19: y(active, starting, since 0.00976175s), standbys: x 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T21:29:16.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T21:29:16.570 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: Manager daemon y is now available 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T21:29:16.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:16 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3665716841' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:29:16.718 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:17.244 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:17.275 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:17.309 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:17.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: [26/Mar/2024:21:29:16] ENGINE Bus STARTING 2024-03-26T21:29:17.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: [26/Mar/2024:21:29:16] ENGINE Serving on https://172.21.15.55:7150 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: mgrmap e20: y(active, since 1.03075s), standbys: x 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: [26/Mar/2024:21:29:16] ENGINE Serving on http://172.21.15.55:8765 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: [26/Mar/2024:21:29:16] ENGINE Bus STARTED 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[29406]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: [26/Mar/2024:21:29:16] ENGINE Bus STARTING 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: [26/Mar/2024:21:29:16] ENGINE Serving on https://172.21.15.55:7150 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: mgrmap e20: y(active, since 1.03075s), standbys: x 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: [26/Mar/2024:21:29:16] ENGINE Serving on http://172.21.15.55:8765 2024-03-26T21:29:17.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: [26/Mar/2024:21:29:16] ENGINE Bus STARTED 2024-03-26T21:29:17.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:17.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:17 smithi055 ceph-mon[40587]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: [26/Mar/2024:21:29:16] ENGINE Bus STARTING 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: [26/Mar/2024:21:29:16] ENGINE Serving on https://172.21.15.55:7150 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: mgrmap e20: y(active, since 1.03075s), standbys: x 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: [26/Mar/2024:21:29:16] ENGINE Serving on http://172.21.15.55:8765 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: [26/Mar/2024:21:29:16] ENGINE Bus STARTED 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:17.581 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:17 smithi067 ceph-mon[33346]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T21:29:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:18 smithi067 ceph-mon[33346]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:18 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:18 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:18 smithi067 ceph-mon[33346]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:29:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[40587]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[40587]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[29406]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-mon[29406]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T21:29:19.303 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:29:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[29406]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:29:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[40587]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:29:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:19 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:19 smithi067 ceph-mon[33346]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T21:29:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:19 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:19 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:29:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 3760M 2024-03-26T21:29:20.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:20 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.784 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 3760M 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.785 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:20.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 3760M 2024-03-26T21:29:20.786 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:20 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:22.956 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:22 smithi055 ceph-mon[29406]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T21:29:22.957 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:22 smithi055 ceph-mon[40587]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T21:29:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:22 smithi067 ceph-mon[33346]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T21:29:24.755 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:24 smithi055 ceph-mon[40587]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:29:24.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:24 smithi055 ceph-mon[29406]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:29:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:24 smithi067 ceph-mon[33346]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T21:29:25.050 INFO:teuthology.orchestra.run.smithi055.stdout:253403070477 2024-03-26T21:29:25.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T21:29:25.793 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:25.793 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:25.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:25.793 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:25.823 INFO:teuthology.orchestra.run.smithi055.stdout:176093659159 2024-03-26T21:29:25.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T21:29:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:26.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:26 smithi067 ceph-mon[33346]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:29:26.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:26 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:26 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[40587]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:29:26.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[29406]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T21:29:26.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:26.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:26 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 3761M 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:27.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[40587]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 3761M 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:27.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:27 smithi055 ceph-mon[29406]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 3761M 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: Updating smithi055:/etc/ceph/ceph.conf 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: Updating smithi067:/etc/ceph/ceph.conf 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:27.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:27 smithi067 ceph-mon[33346]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T21:29:28.484 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:28.516 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:28.914 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.915 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[29406]: Deploying daemon alertmanager.a on smithi055 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-mon[40587]: Deploying daemon alertmanager.a on smithi055 2024-03-26T21:29:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: Updating smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: Updating smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:28 smithi067 ceph-mon[33346]: Deploying daemon alertmanager.a on smithi055 2024-03-26T21:29:29.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:29:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:30.504 INFO:teuthology.orchestra.run.smithi055.stdout:133143986206 2024-03-26T21:29:30.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T21:29:30.752 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[40587]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[29406]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:30.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:30 smithi067 ceph-mon[33346]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T21:29:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:31.481 INFO:teuthology.orchestra.run.smithi055.stdout:77309411369 2024-03-26T21:29:31.481 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T21:29:31.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:31 smithi055 ceph-mon[40587]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:29:31.916 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:31 smithi055 ceph-mon[40587]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:29:31.916 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:31 smithi055 ceph-mon[29406]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:29:31.916 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:31 smithi055 ceph-mon[29406]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:29:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:31 smithi067 ceph-mon[33346]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T21:29:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:31 smithi067 ceph-mon[33346]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T21:29:32.897 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:32 smithi055 ceph-mon[40587]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:29:32.897 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:32 smithi055 ceph-mon[29406]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:29:32.923 INFO:teuthology.orchestra.run.smithi055.stdout:34359738420 2024-03-26T21:29:32.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T21:29:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:32 smithi067 ceph-mon[33346]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:29:34.837 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:34 smithi055 ceph-mon[40587]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:34.838 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:34 smithi055 ceph-mon[29406]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:34.878 INFO:teuthology.orchestra.run.smithi055.stdout:55834574896 2024-03-26T21:29:34.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T21:29:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:34 smithi067 ceph-mon[33346]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:35.741 INFO:teuthology.orchestra.run.smithi055.stdout:214748364822 2024-03-26T21:29:35.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T21:29:35.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/783138160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:29:35.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/783138160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:29:35.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/783138160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:29:36.277 INFO:teuthology.orchestra.run.smithi055.stdout:176093659161 2024-03-26T21:29:36.723 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:36 smithi067 ceph-mon[33346]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:36 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:36 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:36 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:29:37.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[29406]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[40587]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:37.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:36 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:29:37.427 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659159 got 176093659161 for osd.5 2024-03-26T21:29:37.427 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:37.578 INFO:teuthology.orchestra.run.smithi055.stdout:111669149732 2024-03-26T21:29:37.578 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T21:29:37.587 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:37.638 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:37.638 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:29:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:29:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:29:37.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:29:37.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: 4.c deep-scrub starts 2024-03-26T21:29:37.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:37 smithi067 ceph-mon[33346]: 4.c deep-scrub ok 2024-03-26T21:29:38.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: 4.c deep-scrub starts 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[29406]: 4.c deep-scrub ok 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:29:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:29:38.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: 4.c deep-scrub starts 2024-03-26T21:29:38.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:37 smithi055 ceph-mon[40587]: 4.c deep-scrub ok 2024-03-26T21:29:38.731 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:38.731 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:38.985 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[40587]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:29:38.985 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:29:38.985 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:29:38.986 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[29406]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:29:38.986 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:29:38.986 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:29:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:38 smithi067 ceph-mon[33346]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:29:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:38 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:29:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:38 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:29:39.242 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:29:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:39.895 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:40.044 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:29:40.044 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:29:40.044 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:29:40.045 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:29:40.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:29:40.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:29:40.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:29:40.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:39 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:29:40.172 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:39 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:29:40.172 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:39 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:29:40.172 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:39 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:29:40.172 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:39 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:29:41.057 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:29:41.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:40 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:29:41.133 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:40 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[40587]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[40587]: 3.8 scrub starts 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[40587]: 3.8 scrub ok 2024-03-26T21:29:42.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:42.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:42.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[29406]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:29:42.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[29406]: 3.8 scrub starts 2024-03-26T21:29:42.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:41 smithi055 ceph-mon[29406]: 3.8 scrub ok 2024-03-26T21:29:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:41 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:41 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:41 smithi067 ceph-mon[33346]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T21:29:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:41 smithi067 ceph-mon[33346]: 3.8 scrub starts 2024-03-26T21:29:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:41 smithi067 ceph-mon[33346]: 3.8 scrub ok 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 3.4 deep-scrub starts 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 3.4 deep-scrub ok 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 6.6 deep-scrub starts 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[40587]: 6.6 deep-scrub ok 2024-03-26T21:29:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 3.4 deep-scrub starts 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 3.4 deep-scrub ok 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 6.6 deep-scrub starts 2024-03-26T21:29:43.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:42 smithi055 ceph-mon[29406]: 6.6 deep-scrub ok 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 3.4 deep-scrub starts 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 3.4 deep-scrub ok 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:29:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:29:43.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 6.6 deep-scrub starts 2024-03-26T21:29:43.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:42 smithi067 ceph-mon[33346]: 6.6 deep-scrub ok 2024-03-26T21:29:43.608 INFO:teuthology.orchestra.run.smithi055.stdout:55834574897 2024-03-26T21:29:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1073229143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:29:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1073229143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:29:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1073229143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:29:44.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574896 got 55834574897 for osd.1 2024-03-26T21:29:44.332 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[40587]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:29:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:29:45.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[29406]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:45.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:29:45.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:44 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:29:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:44 smithi067 ceph-mon[33346]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:44 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:29:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:44 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:29:45.369 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:46.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 3.e deep-scrub starts 2024-03-26T21:29:46.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 3.e deep-scrub ok 2024-03-26T21:29:46.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 5.18 scrub starts 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[40587]: 5.18 scrub ok 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 3.e deep-scrub starts 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 3.e deep-scrub ok 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 5.18 scrub starts 2024-03-26T21:29:46.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:45 smithi055 ceph-mon[29406]: 5.18 scrub ok 2024-03-26T21:29:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 3.e deep-scrub starts 2024-03-26T21:29:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 3.e deep-scrub ok 2024-03-26T21:29:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:29:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:29:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:29:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 5.18 scrub starts 2024-03-26T21:29:46.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:45 smithi067 ceph-mon[33346]: 5.18 scrub ok 2024-03-26T21:29:46.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:47.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[29406]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T21:29:47.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:29:47.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:29:47.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[40587]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T21:29:47.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:29:47.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:46 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:29:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:46 smithi067 ceph-mon[33346]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T21:29:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:46 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:29:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:46 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:29:48.007 INFO:teuthology.orchestra.run.smithi055.stdout:253403070483 2024-03-26T21:29:48.007 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:48.008 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:48.250 INFO:teuthology.orchestra.run.smithi055.stdout:214748364825 2024-03-26T21:29:48.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070477 got 253403070483 for osd.7 2024-03-26T21:29:48.867 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: 5.15 deep-scrub starts 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: 5.15 deep-scrub ok 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/622480656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1372853548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:29:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: 6.2 deep-scrub starts 2024-03-26T21:29:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:48 smithi067 ceph-mon[33346]: 6.2 deep-scrub ok 2024-03-26T21:29:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:29:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: 5.15 deep-scrub starts 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: 5.15 deep-scrub ok 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/622480656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1372853548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: 6.2 deep-scrub starts 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[40587]: 6.2 deep-scrub ok 2024-03-26T21:29:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: 5.15 deep-scrub starts 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: 5.15 deep-scrub ok 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/622480656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1372853548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: 6.2 deep-scrub starts 2024-03-26T21:29:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:48 smithi055 ceph-mon[29406]: 6.2 deep-scrub ok 2024-03-26T21:29:49.777 INFO:teuthology.orchestra.run.smithi055.stdout:133143986210 2024-03-26T21:29:49.903 INFO:teuthology.orchestra.run.smithi055.stdout:34359738424 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/837831995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:29:50.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:29:50.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:29:50.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:29:50.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:29:50.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/837831995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:29:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:29:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:29:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:29:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:29:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:29:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:29:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/837831995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:29:50.391 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364822 got 214748364825 for osd.6 2024-03-26T21:29:50.392 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:50.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:50.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4170465935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:29:50.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:29:50.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:29:50.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:29:50.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:50 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:29:51.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4170465935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4170465935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:29:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:50 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:29:52.009 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:51 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:29:52.009 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:51 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:29:52.010 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:51 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:29:52.010 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:51 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:29:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:51 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:29:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:51 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:29:52.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986210 for osd.4 2024-03-26T21:29:52.465 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[40587]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:29:53.060 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[29406]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T21:29:53.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:29:53.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:29:53.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:29:53.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:52 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:29:53.093 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738424 for osd.0 2024-03-26T21:29:53.093 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:52 smithi067 ceph-mon[33346]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T21:29:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:52 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:29:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:52 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:29:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:52 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:29:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:52 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:29:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:29:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:29:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:29:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:29:53.983 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:29:53.983 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:29:53.983 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:29:53.983 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:53 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:29:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:53 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:29:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:53 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:29:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:53 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:29:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:53 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:29:54.931 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[40587]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:54.931 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:29:54.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:29:54.932 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[29406]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:54.932 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:29:54.932 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:54 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:29:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:54 smithi067 ceph-mon[33346]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:54 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:29:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:54 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:29:55.769 INFO:teuthology.orchestra.run.smithi055.stdout:111669149736 2024-03-26T21:29:56.003 INFO:teuthology.orchestra.run.smithi055.stdout:77309411374 2024-03-26T21:29:56.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:29:56.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:29:56.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 4.7 deep-scrub starts 2024-03-26T21:29:56.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 4.7 deep-scrub ok 2024-03-26T21:29:56.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:29:56.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:29:56.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1992650756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 4.7 deep-scrub starts 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 4.7 deep-scrub ok 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:29:56.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1992650756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 4.7 deep-scrub starts 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 4.7 deep-scrub ok 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:29:56.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1992650756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:29:56.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:29:56.741 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411374 for osd.2 2024-03-26T21:29:56.741 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:29:57.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 4.5 deep-scrub starts 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 4.5 deep-scrub ok 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1115243394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:29:57.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:29:57.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 4.5 deep-scrub starts 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 4.5 deep-scrub ok 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1115243394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:29:57.070 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:56 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:29:57.150 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149732 got 111669149736 for osd.3 2024-03-26T21:29:57.150 DEBUG:teuthology.parallel:result is None 2024-03-26T21:29:57.151 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T21:29:57.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:29:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:29:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 4.5 deep-scrub starts 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 4.5 deep-scrub ok 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1115243394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:29:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:56 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:29:57.407 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 podman[75184]: 2024-03-26 21:29:57.222834443 +0000 UTC m=+0.449473736 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T21:29:57.407 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 podman[75184]: 2024-03-26 21:29:57.414805329 +0000 UTC m=+0.641444604 container create e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:57.858 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:57.981 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:57.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:57.982 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 podman[75184]: 2024-03-26 21:29:57.717742141 +0000 UTC m=+0.944381417 container init e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:57.982 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 podman[75184]: 2024-03-26 21:29:57.720636721 +0000 UTC m=+0.947275997 container start e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.733Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.733Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.734Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.55 port=9094 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.735Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.759Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.759Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.762Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:57.762Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 bash[75184]: e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf 2024-03-26T21:29:57.983 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:57 smithi055 systemd[1]: Started Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:29:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:29:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:29:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:29:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:29:58.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:29:58.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:29:58.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:29:58.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:58.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:57 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:58.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE Bus STOPPING 2024-03-26T21:29:58.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:29:58.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE Bus STOPPED 2024-03-26T21:29:58.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE Bus STARTING 2024-03-26T21:29:58.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE Serving on http://:::9283 2024-03-26T21:29:58.298 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:29:58] ENGINE Bus STARTED 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: Deploying daemon grafana.a on smithi067 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:29:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:58 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:29:59.288 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:29:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:29:59.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: Deploying daemon grafana.a on smithi067 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:29:59.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: Deploying daemon grafana.a on smithi067 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:29:59.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:58 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:29:59.937 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:29:59.736Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000440541s 2024-03-26T21:30:00.072 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:30:00.074 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:30:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:29:59 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:30:00.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:30:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:30:00.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:29:59 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:30:00.875 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-26T21:29:59.279566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1009,"num_read_kb":855,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240716,"kb_used_data":7944,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720500,"statfs":{"total":767960285184,"available":767713792000,"internally_reserved":0,"allocated":8134656,"data_stored":4544655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002054"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771079+0000","last_change":"2024-03-26T21:28:40.880308+0000","last_active":"2024-03-26T21:29:40.771079+0000","last_peered":"2024-03-26T21:29:40.771079+0000","last_clean":"2024-03-26T21:29:40.771079+0000","last_became_active":"2024-03-26T21:28:40.879971+0000","last_became_peered":"2024-03-26T21:28:40.879971+0000","last_unstale":"2024-03-26T21:29:40.771079+0000","last_undegraded":"2024-03-26T21:29:40.771079+0000","last_fullsized":"2024-03-26T21:29:40.771079+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:52.156812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889756+0000","last_change":"2024-03-26T21:28:38.866679+0000","last_active":"2024-03-26T21:29:55.889756+0000","last_peered":"2024-03-26T21:29:55.889756+0000","last_clean":"2024-03-26T21:29:55.889756+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:29:55.889756+0000","last_undegraded":"2024-03-26T21:29:55.889756+0000","last_fullsized":"2024-03-26T21:29:55.889756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:03.904348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890820+0000","last_change":"2024-03-26T21:28:34.844384+0000","last_active":"2024-03-26T21:29:55.890820+0000","last_peered":"2024-03-26T21:29:55.890820+0000","last_clean":"2024-03-26T21:29:55.890820+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:29:55.890820+0000","last_undegraded":"2024-03-26T21:29:55.890820+0000","last_fullsized":"2024-03-26T21:29:55.890820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:51.544689+0000","last_change":"2024-03-26T21:29:51.544731+0000","last_active":"2024-03-26T21:29:51.544689+0000","last_peered":"2024-03-26T21:29:51.544689+0000","last_clean":"2024-03-26T21:29:51.544689+0000","last_became_active":"2024-03-26T21:28:36.841677+0000","last_became_peered":"2024-03-26T21:28:36.841677+0000","last_unstale":"2024-03-26T21:29:51.544689+0000","last_undegraded":"2024-03-26T21:29:51.544689+0000","last_fullsized":"2024-03-26T21:29:51.544689+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.477780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.749455+0000","last_change":"2024-03-26T21:29:49.749475+0000","last_active":"2024-03-26T21:29:49.749455+0000","last_peered":"2024-03-26T21:29:49.749455+0000","last_clean":"2024-03-26T21:29:49.749455+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:29:49.749455+0000","last_undegraded":"2024-03-26T21:29:49.749455+0000","last_fullsized":"2024-03-26T21:29:49.749455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:49.749386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:49.749386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.347098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:45.645645+0000","last_change":"2024-03-26T21:29:45.645672+0000","last_active":"2024-03-26T21:29:45.645645+0000","last_peered":"2024-03-26T21:29:45.645645+0000","last_clean":"2024-03-26T21:29:45.645645+0000","last_became_active":"2024-03-26T21:28:38.866501+0000","last_became_peered":"2024-03-26T21:28:38.866501+0000","last_unstale":"2024-03-26T21:29:45.645645+0000","last_undegraded":"2024-03-26T21:29:45.645645+0000","last_fullsized":"2024-03-26T21:29:45.645645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.799670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:50.735259+0000","last_change":"2024-03-26T21:29:50.735279+0000","last_active":"2024-03-26T21:29:50.735259+0000","last_peered":"2024-03-26T21:29:50.735259+0000","last_clean":"2024-03-26T21:29:50.735259+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:29:50.735259+0000","last_undegraded":"2024-03-26T21:29:50.735259+0000","last_fullsized":"2024-03-26T21:29:50.735259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.735199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.735199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.656417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775649+0000","last_change":"2024-03-26T21:28:36.846364+0000","last_active":"2024-03-26T21:29:40.775649+0000","last_peered":"2024-03-26T21:29:40.775649+0000","last_clean":"2024-03-26T21:29:40.775649+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:29:40.775649+0000","last_undegraded":"2024-03-26T21:29:40.775649+0000","last_fullsized":"2024-03-26T21:29:40.775649+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:47.364815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889359+0000","last_change":"2024-03-26T21:28:40.888299+0000","last_active":"2024-03-26T21:29:55.889359+0000","last_peered":"2024-03-26T21:29:55.889359+0000","last_clean":"2024-03-26T21:29:55.889359+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:29:55.889359+0000","last_undegraded":"2024-03-26T21:29:55.889359+0000","last_fullsized":"2024-03-26T21:29:55.889359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890581+0000","last_change":"2024-03-26T21:28:38.872667+0000","last_active":"2024-03-26T21:29:55.890581+0000","last_peered":"2024-03-26T21:29:55.890581+0000","last_clean":"2024-03-26T21:29:55.890581+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:29:55.890581+0000","last_undegraded":"2024-03-26T21:29:55.890581+0000","last_fullsized":"2024-03-26T21:29:55.890581+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.510486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890783+0000","last_change":"2024-03-26T21:28:34.842302+0000","last_active":"2024-03-26T21:29:55.890783+0000","last_peered":"2024-03-26T21:29:55.890783+0000","last_clean":"2024-03-26T21:29:55.890783+0000","last_became_active":"2024-03-26T21:28:34.841544+0000","last_became_peered":"2024-03-26T21:28:34.841544+0000","last_unstale":"2024-03-26T21:29:55.890783+0000","last_undegraded":"2024-03-26T21:29:55.890783+0000","last_fullsized":"2024-03-26T21:29:55.890783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.708588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:52.496798+0000","last_change":"2024-03-26T21:29:52.496860+0000","last_active":"2024-03-26T21:29:52.496798+0000","last_peered":"2024-03-26T21:29:52.496798+0000","last_clean":"2024-03-26T21:29:52.496798+0000","last_became_active":"2024-03-26T21:28:36.847804+0000","last_became_peered":"2024-03-26T21:28:36.847804+0000","last_unstale":"2024-03-26T21:29:52.496798+0000","last_undegraded":"2024-03-26T21:29:52.496798+0000","last_fullsized":"2024-03-26T21:29:52.496798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.032728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890297+0000","last_change":"2024-03-26T21:28:40.887065+0000","last_active":"2024-03-26T21:29:55.890297+0000","last_peered":"2024-03-26T21:29:55.890297+0000","last_clean":"2024-03-26T21:29:55.890297+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:29:55.890297+0000","last_undegraded":"2024-03-26T21:29:55.890297+0000","last_fullsized":"2024-03-26T21:29:55.890297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583919+0000","last_change":"2024-03-26T21:28:38.864721+0000","last_active":"2024-03-26T21:29:41.583919+0000","last_peered":"2024-03-26T21:29:41.583919+0000","last_clean":"2024-03-26T21:29:41.583919+0000","last_became_active":"2024-03-26T21:28:38.864406+0000","last_became_peered":"2024-03-26T21:28:38.864406+0000","last_unstale":"2024-03-26T21:29:41.583919+0000","last_undegraded":"2024-03-26T21:29:41.583919+0000","last_fullsized":"2024-03-26T21:29:41.583919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:43.916601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890719+0000","last_change":"2024-03-26T21:28:34.847255+0000","last_active":"2024-03-26T21:29:55.890719+0000","last_peered":"2024-03-26T21:29:55.890719+0000","last_clean":"2024-03-26T21:29:55.890719+0000","last_became_active":"2024-03-26T21:28:34.846956+0000","last_became_peered":"2024-03-26T21:28:34.846956+0000","last_unstale":"2024-03-26T21:29:55.890719+0000","last_undegraded":"2024-03-26T21:29:55.890719+0000","last_fullsized":"2024-03-26T21:29:55.890719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.186750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771559+0000","last_change":"2024-03-26T21:28:36.846687+0000","last_active":"2024-03-26T21:29:40.771559+0000","last_peered":"2024-03-26T21:29:40.771559+0000","last_clean":"2024-03-26T21:29:40.771559+0000","last_became_active":"2024-03-26T21:28:36.846364+0000","last_became_peered":"2024-03-26T21:28:36.846364+0000","last_unstale":"2024-03-26T21:29:40.771559+0000","last_undegraded":"2024-03-26T21:29:40.771559+0000","last_fullsized":"2024-03-26T21:29:40.771559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.374326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772076+0000","last_change":"2024-03-26T21:28:41.114040+0000","last_active":"2024-03-26T21:29:40.772076+0000","last_peered":"2024-03-26T21:29:40.772076+0000","last_clean":"2024-03-26T21:29:40.772076+0000","last_became_active":"2024-03-26T21:28:41.113734+0000","last_became_peered":"2024-03-26T21:28:41.113734+0000","last_unstale":"2024-03-26T21:29:40.772076+0000","last_undegraded":"2024-03-26T21:29:40.772076+0000","last_fullsized":"2024-03-26T21:29:40.772076+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.261763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891379+0000","last_change":"2024-03-26T21:28:38.871499+0000","last_active":"2024-03-26T21:29:55.891379+0000","last_peered":"2024-03-26T21:29:55.891379+0000","last_clean":"2024-03-26T21:29:55.891379+0000","last_became_active":"2024-03-26T21:28:38.870984+0000","last_became_peered":"2024-03-26T21:28:38.870984+0000","last_unstale":"2024-03-26T21:29:55.891379+0000","last_undegraded":"2024-03-26T21:29:55.891379+0000","last_fullsized":"2024-03-26T21:29:55.891379+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891278+0000","last_change":"2024-03-26T21:28:34.847658+0000","last_active":"2024-03-26T21:29:55.891278+0000","last_peered":"2024-03-26T21:29:55.891278+0000","last_clean":"2024-03-26T21:29:55.891278+0000","last_became_active":"2024-03-26T21:28:34.846913+0000","last_became_peered":"2024-03-26T21:28:34.846913+0000","last_unstale":"2024-03-26T21:29:55.891278+0000","last_undegraded":"2024-03-26T21:29:55.891278+0000","last_fullsized":"2024-03-26T21:29:55.891278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:45.654328+0000","last_change":"2024-03-26T21:29:45.654379+0000","last_active":"2024-03-26T21:29:45.654328+0000","last_peered":"2024-03-26T21:29:45.654328+0000","last_clean":"2024-03-26T21:29:45.654328+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:29:45.654328+0000","last_undegraded":"2024-03-26T21:29:45.654328+0000","last_fullsized":"2024-03-26T21:29:45.654328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:29:45.654238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.654238+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.970402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776109+0000","last_change":"2024-03-26T21:28:41.113863+0000","last_active":"2024-03-26T21:29:40.776109+0000","last_peered":"2024-03-26T21:29:40.776109+0000","last_clean":"2024-03-26T21:29:40.776109+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:29:40.776109+0000","last_undegraded":"2024-03-26T21:29:40.776109+0000","last_fullsized":"2024-03-26T21:29:40.776109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.033803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:44.615977+0000","last_change":"2024-03-26T21:29:44.616004+0000","last_active":"2024-03-26T21:29:44.615977+0000","last_peered":"2024-03-26T21:29:44.615977+0000","last_clean":"2024-03-26T21:29:44.615977+0000","last_became_active":"2024-03-26T21:28:38.867256+0000","last_became_peered":"2024-03-26T21:28:38.867256+0000","last_unstale":"2024-03-26T21:29:44.615977+0000","last_undegraded":"2024-03-26T21:29:44.615977+0000","last_fullsized":"2024-03-26T21:29:44.615977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.353334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771290+0000","last_change":"2024-03-26T21:28:34.846901+0000","last_active":"2024-03-26T21:29:40.771290+0000","last_peered":"2024-03-26T21:29:40.771290+0000","last_clean":"2024-03-26T21:29:40.771290+0000","last_became_active":"2024-03-26T21:28:34.846583+0000","last_became_peered":"2024-03-26T21:28:34.846583+0000","last_unstale":"2024-03-26T21:29:40.771290+0000","last_undegraded":"2024-03-26T21:29:40.771290+0000","last_fullsized":"2024-03-26T21:29:40.771290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:53.292403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775827+0000","last_change":"2024-03-26T21:28:36.850322+0000","last_active":"2024-03-26T21:29:40.775827+0000","last_peered":"2024-03-26T21:29:40.775827+0000","last_clean":"2024-03-26T21:29:40.775827+0000","last_became_active":"2024-03-26T21:28:36.849968+0000","last_became_peered":"2024-03-26T21:28:36.849968+0000","last_unstale":"2024-03-26T21:29:40.775827+0000","last_undegraded":"2024-03-26T21:29:40.775827+0000","last_fullsized":"2024-03-26T21:29:40.775827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:54.614142+0000","last_change":"2024-03-26T21:29:54.614182+0000","last_active":"2024-03-26T21:29:54.614142+0000","last_peered":"2024-03-26T21:29:54.614142+0000","last_clean":"2024-03-26T21:29:54.614142+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:29:54.614142+0000","last_undegraded":"2024-03-26T21:29:54.614142+0000","last_fullsized":"2024-03-26T21:29:54.614142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:29:54.614064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:54.614064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:06.934031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899894+0000","last_change":"2024-03-26T21:28:38.867627+0000","last_active":"2024-03-26T21:29:49.899894+0000","last_peered":"2024-03-26T21:29:49.899894+0000","last_clean":"2024-03-26T21:29:49.899894+0000","last_became_active":"2024-03-26T21:28:38.867245+0000","last_became_peered":"2024-03-26T21:28:38.867245+0000","last_unstale":"2024-03-26T21:29:49.899894+0000","last_undegraded":"2024-03-26T21:29:49.899894+0000","last_fullsized":"2024-03-26T21:29:49.899894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.894707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889851+0000","last_change":"2024-03-26T21:28:34.846724+0000","last_active":"2024-03-26T21:29:55.889851+0000","last_peered":"2024-03-26T21:29:55.889851+0000","last_clean":"2024-03-26T21:29:55.889851+0000","last_became_active":"2024-03-26T21:28:34.846408+0000","last_became_peered":"2024-03-26T21:28:34.846408+0000","last_unstale":"2024-03-26T21:29:55.889851+0000","last_undegraded":"2024-03-26T21:29:55.889851+0000","last_fullsized":"2024-03-26T21:29:55.889851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891869+0000","last_change":"2024-03-26T21:28:37.143422+0000","last_active":"2024-03-26T21:29:55.891869+0000","last_peered":"2024-03-26T21:29:55.891869+0000","last_clean":"2024-03-26T21:29:55.891869+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:29:55.891869+0000","last_undegraded":"2024-03-26T21:29:55.891869+0000","last_fullsized":"2024-03-26T21:29:55.891869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.464940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891044+0000","last_change":"2024-03-26T21:29:45.092723+0000","last_active":"2024-03-26T21:29:55.891044+0000","last_peered":"2024-03-26T21:29:55.891044+0000","last_clean":"2024-03-26T21:29:55.891044+0000","last_became_active":"2024-03-26T21:28:40.881781+0000","last_became_peered":"2024-03-26T21:28:40.881781+0000","last_unstale":"2024-03-26T21:29:55.891044+0000","last_undegraded":"2024-03-26T21:29:55.891044+0000","last_fullsized":"2024-03-26T21:29:55.891044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.160967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.892051+0000","last_change":"2024-03-26T21:28:38.867746+0000","last_active":"2024-03-26T21:29:55.892051+0000","last_peered":"2024-03-26T21:29:55.892051+0000","last_clean":"2024-03-26T21:29:55.892051+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:29:55.892051+0000","last_undegraded":"2024-03-26T21:29:55.892051+0000","last_fullsized":"2024-03-26T21:29:55.892051+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775121+0000","last_change":"2024-03-26T21:28:34.849241+0000","last_active":"2024-03-26T21:29:40.775121+0000","last_peered":"2024-03-26T21:29:40.775121+0000","last_clean":"2024-03-26T21:29:40.775121+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:29:40.775121+0000","last_undegraded":"2024-03-26T21:29:40.775121+0000","last_fullsized":"2024-03-26T21:29:40.775121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:45.919934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.864214+0000","last_change":"2024-03-26T21:29:36.821598+0000","last_active":"2024-03-26T21:29:40.864214+0000","last_peered":"2024-03-26T21:29:40.864214+0000","last_clean":"2024-03-26T21:29:40.864214+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:29:40.864214+0000","last_undegraded":"2024-03-26T21:29:40.864214+0000","last_fullsized":"2024-03-26T21:29:40.864214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:29:36.821436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.821436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.432234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583914+0000","last_change":"2024-03-26T21:28:40.885889+0000","last_active":"2024-03-26T21:29:41.583914+0000","last_peered":"2024-03-26T21:29:41.583914+0000","last_clean":"2024-03-26T21:29:41.583914+0000","last_became_active":"2024-03-26T21:28:40.885572+0000","last_became_peered":"2024-03-26T21:28:40.885572+0000","last_unstale":"2024-03-26T21:29:41.583914+0000","last_undegraded":"2024-03-26T21:29:41.583914+0000","last_fullsized":"2024-03-26T21:29:41.583914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:44.002943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.864369+0000","last_change":"2024-03-26T21:28:38.870467+0000","last_active":"2024-03-26T21:29:40.864369+0000","last_peered":"2024-03-26T21:29:40.864369+0000","last_clean":"2024-03-26T21:29:40.864369+0000","last_became_active":"2024-03-26T21:28:38.870142+0000","last_became_peered":"2024-03-26T21:28:38.870142+0000","last_unstale":"2024-03-26T21:29:40.864369+0000","last_undegraded":"2024-03-26T21:29:40.864369+0000","last_fullsized":"2024-03-26T21:29:40.864369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890725+0000","last_change":"2024-03-26T21:29:52.893256+0000","last_active":"2024-03-26T21:29:55.890725+0000","last_peered":"2024-03-26T21:29:55.890725+0000","last_clean":"2024-03-26T21:29:55.890725+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:29:55.890725+0000","last_undegraded":"2024-03-26T21:29:55.890725+0000","last_fullsized":"2024-03-26T21:29:55.890725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.893078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.893078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.394353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775121+0000","last_change":"2024-03-26T21:28:36.840820+0000","last_active":"2024-03-26T21:29:40.775121+0000","last_peered":"2024-03-26T21:29:40.775121+0000","last_clean":"2024-03-26T21:29:40.775121+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:29:40.775121+0000","last_undegraded":"2024-03-26T21:29:40.775121+0000","last_fullsized":"2024-03-26T21:29:40.775121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:44.639728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775387+0000","last_change":"2024-03-26T21:28:41.117667+0000","last_active":"2024-03-26T21:29:40.775387+0000","last_peered":"2024-03-26T21:29:40.775387+0000","last_clean":"2024-03-26T21:29:40.775387+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:29:40.775387+0000","last_undegraded":"2024-03-26T21:29:40.775387+0000","last_fullsized":"2024-03-26T21:29:40.775387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.963275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584112+0000","last_change":"2024-03-26T21:28:38.864935+0000","last_active":"2024-03-26T21:29:41.584112+0000","last_peered":"2024-03-26T21:29:41.584112+0000","last_clean":"2024-03-26T21:29:41.584112+0000","last_became_active":"2024-03-26T21:28:38.864421+0000","last_became_peered":"2024-03-26T21:28:38.864421+0000","last_unstale":"2024-03-26T21:29:41.584112+0000","last_undegraded":"2024-03-26T21:29:41.584112+0000","last_fullsized":"2024-03-26T21:29:41.584112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:53.514362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891131+0000","last_change":"2024-03-26T21:28:34.847955+0000","last_active":"2024-03-26T21:29:55.891131+0000","last_peered":"2024-03-26T21:29:55.891131+0000","last_clean":"2024-03-26T21:29:55.891131+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:29:55.891131+0000","last_undegraded":"2024-03-26T21:29:55.891131+0000","last_fullsized":"2024-03-26T21:29:55.891131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775545+0000","last_change":"2024-03-26T21:28:37.142935+0000","last_active":"2024-03-26T21:29:40.775545+0000","last_peered":"2024-03-26T21:29:40.775545+0000","last_clean":"2024-03-26T21:29:40.775545+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:29:40.775545+0000","last_undegraded":"2024-03-26T21:29:40.775545+0000","last_fullsized":"2024-03-26T21:29:40.775545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.315521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889091+0000","last_change":"2024-03-26T21:28:40.889084+0000","last_active":"2024-03-26T21:29:55.889091+0000","last_peered":"2024-03-26T21:29:55.889091+0000","last_clean":"2024-03-26T21:29:55.889091+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:29:55.889091+0000","last_undegraded":"2024-03-26T21:29:55.889091+0000","last_fullsized":"2024-03-26T21:29:55.889091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.980105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776180+0000","last_change":"2024-03-26T21:28:38.865685+0000","last_active":"2024-03-26T21:29:40.776180+0000","last_peered":"2024-03-26T21:29:40.776180+0000","last_clean":"2024-03-26T21:29:40.776180+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:29:40.776180+0000","last_undegraded":"2024-03-26T21:29:40.776180+0000","last_fullsized":"2024-03-26T21:29:40.776180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:46.796618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.577570+0000","last_change":"2024-03-26T21:29:55.577602+0000","last_active":"2024-03-26T21:29:55.577570+0000","last_peered":"2024-03-26T21:29:55.577570+0000","last_clean":"2024-03-26T21:29:55.577570+0000","last_became_active":"2024-03-26T21:28:34.848518+0000","last_became_peered":"2024-03-26T21:28:34.848518+0000","last_unstale":"2024-03-26T21:29:55.577570+0000","last_undegraded":"2024-03-26T21:29:55.577570+0000","last_fullsized":"2024-03-26T21:29:55.577570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.145785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890534+0000","last_change":"2024-03-26T21:28:36.841238+0000","last_active":"2024-03-26T21:29:55.890534+0000","last_peered":"2024-03-26T21:29:55.890534+0000","last_clean":"2024-03-26T21:29:55.890534+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:29:55.890534+0000","last_undegraded":"2024-03-26T21:29:55.890534+0000","last_fullsized":"2024-03-26T21:29:55.890534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.343773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775168+0000","last_change":"2024-03-26T21:28:40.891718+0000","last_active":"2024-03-26T21:29:40.775168+0000","last_peered":"2024-03-26T21:29:40.775168+0000","last_clean":"2024-03-26T21:29:40.775168+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:29:40.775168+0000","last_undegraded":"2024-03-26T21:29:40.775168+0000","last_fullsized":"2024-03-26T21:29:40.775168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.486662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890351+0000","last_change":"2024-03-26T21:28:38.868625+0000","last_active":"2024-03-26T21:29:55.890351+0000","last_peered":"2024-03-26T21:29:55.890351+0000","last_clean":"2024-03-26T21:29:55.890351+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:29:55.890351+0000","last_undegraded":"2024-03-26T21:29:55.890351+0000","last_fullsized":"2024-03-26T21:29:55.890351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.070691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771704+0000","last_change":"2024-03-26T21:28:34.844448+0000","last_active":"2024-03-26T21:29:40.771704+0000","last_peered":"2024-03-26T21:29:40.771704+0000","last_clean":"2024-03-26T21:29:40.771704+0000","last_became_active":"2024-03-26T21:28:34.842160+0000","last_became_peered":"2024-03-26T21:28:34.842160+0000","last_unstale":"2024-03-26T21:29:40.771704+0000","last_undegraded":"2024-03-26T21:29:40.771704+0000","last_fullsized":"2024-03-26T21:29:40.771704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.941135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771507+0000","last_change":"2024-03-26T21:28:36.846742+0000","last_active":"2024-03-26T21:29:40.771507+0000","last_peered":"2024-03-26T21:29:40.771507+0000","last_clean":"2024-03-26T21:29:40.771507+0000","last_became_active":"2024-03-26T21:28:36.846289+0000","last_became_peered":"2024-03-26T21:28:36.846289+0000","last_unstale":"2024-03-26T21:29:40.771507+0000","last_undegraded":"2024-03-26T21:29:40.771507+0000","last_fullsized":"2024-03-26T21:29:40.771507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:59.376267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891634+0000","last_change":"2024-03-26T21:28:40.891431+0000","last_active":"2024-03-26T21:29:55.891634+0000","last_peered":"2024-03-26T21:29:55.891634+0000","last_clean":"2024-03-26T21:29:55.891634+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:29:55.891634+0000","last_undegraded":"2024-03-26T21:29:55.891634+0000","last_fullsized":"2024-03-26T21:29:55.891634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891087+0000","last_change":"2024-03-26T21:29:47.419090+0000","last_active":"2024-03-26T21:29:55.891087+0000","last_peered":"2024-03-26T21:29:55.891087+0000","last_clean":"2024-03-26T21:29:55.891087+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:29:55.891087+0000","last_undegraded":"2024-03-26T21:29:55.891087+0000","last_fullsized":"2024-03-26T21:29:55.891087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:29:47.418949+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.219762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584045+0000","last_change":"2024-03-26T21:29:39.550921+0000","last_active":"2024-03-26T21:29:41.584045+0000","last_peered":"2024-03-26T21:29:41.584045+0000","last_clean":"2024-03-26T21:29:41.584045+0000","last_became_active":"2024-03-26T21:28:34.839491+0000","last_became_peered":"2024-03-26T21:28:34.839491+0000","last_unstale":"2024-03-26T21:29:41.584045+0000","last_undegraded":"2024-03-26T21:29:41.584045+0000","last_fullsized":"2024-03-26T21:29:41.584045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.089443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.733967+0000","last_change":"2024-03-26T21:29:55.733996+0000","last_active":"2024-03-26T21:29:55.733967+0000","last_peered":"2024-03-26T21:29:55.733967+0000","last_clean":"2024-03-26T21:29:55.733967+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:29:55.733967+0000","last_undegraded":"2024-03-26T21:29:55.733967+0000","last_fullsized":"2024-03-26T21:29:55.733967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:29:55.733905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.733905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.846036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772065+0000","last_change":"2024-03-26T21:28:40.881531+0000","last_active":"2024-03-26T21:29:40.772065+0000","last_peered":"2024-03-26T21:29:40.772065+0000","last_clean":"2024-03-26T21:29:40.772065+0000","last_became_active":"2024-03-26T21:28:40.881234+0000","last_became_peered":"2024-03-26T21:28:40.881234+0000","last_unstale":"2024-03-26T21:29:40.772065+0000","last_undegraded":"2024-03-26T21:29:40.772065+0000","last_fullsized":"2024-03-26T21:29:40.772065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.460159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899751+0000","last_change":"2024-03-26T21:28:38.864606+0000","last_active":"2024-03-26T21:29:49.899751+0000","last_peered":"2024-03-26T21:29:49.899751+0000","last_clean":"2024-03-26T21:29:49.899751+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:29:49.899751+0000","last_undegraded":"2024-03-26T21:29:49.899751+0000","last_fullsized":"2024-03-26T21:29:49.899751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.571305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890177+0000","last_change":"2024-03-26T21:28:34.841082+0000","last_active":"2024-03-26T21:29:55.890177+0000","last_peered":"2024-03-26T21:29:55.890177+0000","last_clean":"2024-03-26T21:29:55.890177+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:29:55.890177+0000","last_undegraded":"2024-03-26T21:29:55.890177+0000","last_fullsized":"2024-03-26T21:29:55.890177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.847025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891139+0000","last_change":"2024-03-26T21:28:37.142871+0000","last_active":"2024-03-26T21:29:55.891139+0000","last_peered":"2024-03-26T21:29:55.891139+0000","last_clean":"2024-03-26T21:29:55.891139+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:29:55.891139+0000","last_undegraded":"2024-03-26T21:29:55.891139+0000","last_fullsized":"2024-03-26T21:29:55.891139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071192+0000","last_change":"2024-03-26T21:28:40.885442+0000","last_active":"2024-03-26T21:29:41.071192+0000","last_peered":"2024-03-26T21:29:41.071192+0000","last_clean":"2024-03-26T21:29:41.071192+0000","last_became_active":"2024-03-26T21:28:40.884963+0000","last_became_peered":"2024-03-26T21:28:40.884963+0000","last_unstale":"2024-03-26T21:29:41.071192+0000","last_undegraded":"2024-03-26T21:29:41.071192+0000","last_fullsized":"2024-03-26T21:29:41.071192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.759989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776431+0000","last_change":"2024-03-26T21:28:38.864418+0000","last_active":"2024-03-26T21:29:40.776431+0000","last_peered":"2024-03-26T21:29:40.776431+0000","last_clean":"2024-03-26T21:29:40.776431+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:29:40.776431+0000","last_undegraded":"2024-03-26T21:29:40.776431+0000","last_fullsized":"2024-03-26T21:29:40.776431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.783527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583901+0000","last_change":"2024-03-26T21:28:34.843600+0000","last_active":"2024-03-26T21:29:41.583901+0000","last_peered":"2024-03-26T21:29:41.583901+0000","last_clean":"2024-03-26T21:29:41.583901+0000","last_became_active":"2024-03-26T21:28:34.843175+0000","last_became_peered":"2024-03-26T21:28:34.843175+0000","last_unstale":"2024-03-26T21:29:41.583901+0000","last_undegraded":"2024-03-26T21:29:41.583901+0000","last_fullsized":"2024-03-26T21:29:41.583901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.404783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891979+0000","last_change":"2024-03-26T21:28:37.143400+0000","last_active":"2024-03-26T21:29:55.891979+0000","last_peered":"2024-03-26T21:29:55.891979+0000","last_clean":"2024-03-26T21:29:55.891979+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:29:55.891979+0000","last_undegraded":"2024-03-26T21:29:55.891979+0000","last_fullsized":"2024-03-26T21:29:55.891979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584291+0000","last_change":"2024-03-26T21:28:41.115111+0000","last_active":"2024-03-26T21:29:41.584291+0000","last_peered":"2024-03-26T21:29:41.584291+0000","last_clean":"2024-03-26T21:29:41.584291+0000","last_became_active":"2024-03-26T21:28:41.114831+0000","last_became_peered":"2024-03-26T21:28:41.114831+0000","last_unstale":"2024-03-26T21:29:41.584291+0000","last_undegraded":"2024-03-26T21:29:41.584291+0000","last_fullsized":"2024-03-26T21:29:41.584291+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.343661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891173+0000","last_change":"2024-03-26T21:28:38.875070+0000","last_active":"2024-03-26T21:29:55.891173+0000","last_peered":"2024-03-26T21:29:55.891173+0000","last_clean":"2024-03-26T21:29:55.891173+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:29:55.891173+0000","last_undegraded":"2024-03-26T21:29:55.891173+0000","last_fullsized":"2024-03-26T21:29:55.891173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:53.937475+0000","last_change":"2024-03-26T21:29:53.937502+0000","last_active":"2024-03-26T21:29:53.937475+0000","last_peered":"2024-03-26T21:29:53.937475+0000","last_clean":"2024-03-26T21:29:53.937475+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:29:53.937475+0000","last_undegraded":"2024-03-26T21:29:53.937475+0000","last_fullsized":"2024-03-26T21:29:53.937475+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:53.937386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:53.937386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.079593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776367+0000","last_change":"2024-03-26T21:28:59.687904+0000","last_active":"2024-03-26T21:29:40.776367+0000","last_peered":"2024-03-26T21:29:40.776367+0000","last_clean":"2024-03-26T21:29:40.776367+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:29:40.776367+0000","last_undegraded":"2024-03-26T21:29:40.776367+0000","last_fullsized":"2024-03-26T21:29:40.776367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:47.645966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890758+0000","last_change":"2024-03-26T21:28:41.114110+0000","last_active":"2024-03-26T21:29:55.890758+0000","last_peered":"2024-03-26T21:29:55.890758+0000","last_clean":"2024-03-26T21:29:55.890758+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:29:55.890758+0000","last_undegraded":"2024-03-26T21:29:55.890758+0000","last_fullsized":"2024-03-26T21:29:55.890758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.352080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584544+0000","last_change":"2024-03-26T21:28:34.843004+0000","last_active":"2024-03-26T21:29:41.584544+0000","last_peered":"2024-03-26T21:29:41.584544+0000","last_clean":"2024-03-26T21:29:41.584544+0000","last_became_active":"2024-03-26T21:28:34.842712+0000","last_became_peered":"2024-03-26T21:28:34.842712+0000","last_unstale":"2024-03-26T21:29:41.584544+0000","last_undegraded":"2024-03-26T21:29:41.584544+0000","last_fullsized":"2024-03-26T21:29:41.584544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.585264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890307+0000","last_change":"2024-03-26T21:28:37.141762+0000","last_active":"2024-03-26T21:29:55.890307+0000","last_peered":"2024-03-26T21:29:55.890307+0000","last_clean":"2024-03-26T21:29:55.890307+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:55.890307+0000","last_undegraded":"2024-03-26T21:29:55.890307+0000","last_fullsized":"2024-03-26T21:29:55.890307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899871+0000","last_change":"2024-03-26T21:28:38.874954+0000","last_active":"2024-03-26T21:29:49.899871+0000","last_peered":"2024-03-26T21:29:49.899871+0000","last_clean":"2024-03-26T21:29:49.899871+0000","last_became_active":"2024-03-26T21:28:38.874646+0000","last_became_peered":"2024-03-26T21:28:38.874646+0000","last_unstale":"2024-03-26T21:29:49.899871+0000","last_undegraded":"2024-03-26T21:29:49.899871+0000","last_fullsized":"2024-03-26T21:29:49.899871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.073289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775762+0000","last_change":"2024-03-26T21:28:40.888897+0000","last_active":"2024-03-26T21:29:40.775762+0000","last_peered":"2024-03-26T21:29:40.775762+0000","last_clean":"2024-03-26T21:29:40.775762+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:29:40.775762+0000","last_undegraded":"2024-03-26T21:29:40.775762+0000","last_fullsized":"2024-03-26T21:29:40.775762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:08.336674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:44.612784+0000","last_change":"2024-03-26T21:29:44.612815+0000","last_active":"2024-03-26T21:29:44.612784+0000","last_peered":"2024-03-26T21:29:44.612784+0000","last_clean":"2024-03-26T21:29:44.612784+0000","last_became_active":"2024-03-26T21:28:34.848110+0000","last_became_peered":"2024-03-26T21:28:34.848110+0000","last_unstale":"2024-03-26T21:29:44.612784+0000","last_undegraded":"2024-03-26T21:29:44.612784+0000","last_fullsized":"2024-03-26T21:29:44.612784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.133060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:42.608528+0000","last_change":"2024-03-26T21:29:42.608574+0000","last_active":"2024-03-26T21:29:42.608528+0000","last_peered":"2024-03-26T21:29:42.608528+0000","last_clean":"2024-03-26T21:29:42.608528+0000","last_became_active":"2024-03-26T21:28:36.847905+0000","last_became_peered":"2024-03-26T21:28:36.847905+0000","last_unstale":"2024-03-26T21:29:42.608528+0000","last_undegraded":"2024-03-26T21:29:42.608528+0000","last_fullsized":"2024-03-26T21:29:42.608528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.811581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.070727+0000","last_change":"2024-03-26T21:28:38.867409+0000","last_active":"2024-03-26T21:29:41.070727+0000","last_peered":"2024-03-26T21:29:41.070727+0000","last_clean":"2024-03-26T21:29:41.070727+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:29:41.070727+0000","last_undegraded":"2024-03-26T21:29:41.070727+0000","last_fullsized":"2024-03-26T21:29:41.070727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:54.845468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583761+0000","last_change":"2024-03-26T21:28:40.887205+0000","last_active":"2024-03-26T21:29:41.583761+0000","last_peered":"2024-03-26T21:29:41.583761+0000","last_clean":"2024-03-26T21:29:41.583761+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:29:41.583761+0000","last_undegraded":"2024-03-26T21:29:41.583761+0000","last_fullsized":"2024-03-26T21:29:41.583761+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.254459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071013+0000","last_change":"2024-03-26T21:28:38.869650+0000","last_active":"2024-03-26T21:29:41.071013+0000","last_peered":"2024-03-26T21:29:41.071013+0000","last_clean":"2024-03-26T21:29:41.071013+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:29:41.071013+0000","last_undegraded":"2024-03-26T21:29:41.071013+0000","last_fullsized":"2024-03-26T21:29:41.071013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.021948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.625690+0000","last_change":"2024-03-26T21:29:48.625718+0000","last_active":"2024-03-26T21:29:48.625690+0000","last_peered":"2024-03-26T21:29:48.625690+0000","last_clean":"2024-03-26T21:29:48.625690+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:29:48.625690+0000","last_undegraded":"2024-03-26T21:29:48.625690+0000","last_fullsized":"2024-03-26T21:29:48.625690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.625595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.625595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.303805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.864700+0000","last_change":"2024-03-26T21:28:37.142550+0000","last_active":"2024-03-26T21:29:40.864700+0000","last_peered":"2024-03-26T21:29:40.864700+0000","last_clean":"2024-03-26T21:29:40.864700+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:29:40.864700+0000","last_undegraded":"2024-03-26T21:29:40.864700+0000","last_fullsized":"2024-03-26T21:29:40.864700+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:54.313162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891027+0000","last_change":"2024-03-26T21:28:40.889290+0000","last_active":"2024-03-26T21:29:55.891027+0000","last_peered":"2024-03-26T21:29:55.891027+0000","last_clean":"2024-03-26T21:29:55.891027+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:29:55.891027+0000","last_undegraded":"2024-03-26T21:29:55.891027+0000","last_fullsized":"2024-03-26T21:29:55.891027+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.738713+0000","last_change":"2024-03-26T21:29:48.738742+0000","last_active":"2024-03-26T21:29:48.738713+0000","last_peered":"2024-03-26T21:29:48.738713+0000","last_clean":"2024-03-26T21:29:48.738713+0000","last_became_active":"2024-03-26T21:28:38.866104+0000","last_became_peered":"2024-03-26T21:28:38.866104+0000","last_unstale":"2024-03-26T21:29:48.738713+0000","last_undegraded":"2024-03-26T21:29:48.738713+0000","last_fullsized":"2024-03-26T21:29:48.738713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.530253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890845+0000","last_change":"2024-03-26T21:28:34.841686+0000","last_active":"2024-03-26T21:29:55.890845+0000","last_peered":"2024-03-26T21:29:55.890845+0000","last_clean":"2024-03-26T21:29:55.890845+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:29:55.890845+0000","last_undegraded":"2024-03-26T21:29:55.890845+0000","last_fullsized":"2024-03-26T21:29:55.890845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.267735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891511+0000","last_change":"2024-03-26T21:28:36.845693+0000","last_active":"2024-03-26T21:29:55.891511+0000","last_peered":"2024-03-26T21:29:55.891511+0000","last_clean":"2024-03-26T21:29:55.891511+0000","last_became_active":"2024-03-26T21:28:36.845358+0000","last_became_peered":"2024-03-26T21:28:36.845358+0000","last_unstale":"2024-03-26T21:29:55.891511+0000","last_undegraded":"2024-03-26T21:29:55.891511+0000","last_fullsized":"2024-03-26T21:29:55.891511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.265401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:42.726393+0000","last_change":"2024-03-26T21:29:42.726419+0000","last_active":"2024-03-26T21:29:42.726393+0000","last_peered":"2024-03-26T21:29:42.726393+0000","last_clean":"2024-03-26T21:29:42.726393+0000","last_became_active":"2024-03-26T21:28:40.884694+0000","last_became_peered":"2024-03-26T21:28:40.884694+0000","last_unstale":"2024-03-26T21:29:42.726393+0000","last_undegraded":"2024-03-26T21:29:42.726393+0000","last_fullsized":"2024-03-26T21:29:42.726393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.138620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890468+0000","last_change":"2024-03-26T21:28:38.867924+0000","last_active":"2024-03-26T21:29:55.890468+0000","last_peered":"2024-03-26T21:29:55.890468+0000","last_clean":"2024-03-26T21:29:55.890468+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:29:55.890468+0000","last_undegraded":"2024-03-26T21:29:55.890468+0000","last_fullsized":"2024-03-26T21:29:55.890468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771400+0000","last_change":"2024-03-26T21:28:34.842465+0000","last_active":"2024-03-26T21:29:40.771400+0000","last_peered":"2024-03-26T21:29:40.771400+0000","last_clean":"2024-03-26T21:29:40.771400+0000","last_became_active":"2024-03-26T21:28:34.842130+0000","last_became_peered":"2024-03-26T21:28:34.842130+0000","last_unstale":"2024-03-26T21:29:40.771400+0000","last_undegraded":"2024-03-26T21:29:40.771400+0000","last_fullsized":"2024-03-26T21:29:40.771400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.625559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890594+0000","last_change":"2024-03-26T21:28:59.692512+0000","last_active":"2024-03-26T21:29:55.890594+0000","last_peered":"2024-03-26T21:29:55.890594+0000","last_clean":"2024-03-26T21:29:55.890594+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:29:55.890594+0000","last_undegraded":"2024-03-26T21:29:55.890594+0000","last_fullsized":"2024-03-26T21:29:55.890594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890365+0000","last_change":"2024-03-26T21:28:38.873506+0000","last_active":"2024-03-26T21:29:55.890365+0000","last_peered":"2024-03-26T21:29:55.890365+0000","last_clean":"2024-03-26T21:29:55.890365+0000","last_became_active":"2024-03-26T21:28:38.873196+0000","last_became_peered":"2024-03-26T21:28:38.873196+0000","last_unstale":"2024-03-26T21:29:55.890365+0000","last_undegraded":"2024-03-26T21:29:55.890365+0000","last_fullsized":"2024-03-26T21:29:55.890365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584559+0000","last_change":"2024-03-26T21:28:41.114073+0000","last_active":"2024-03-26T21:29:41.584559+0000","last_peered":"2024-03-26T21:29:41.584559+0000","last_clean":"2024-03-26T21:29:41.584559+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:29:41.584559+0000","last_undegraded":"2024-03-26T21:29:41.584559+0000","last_fullsized":"2024-03-26T21:29:41.584559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:46.228003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889543+0000","last_change":"2024-03-26T21:29:36.249432+0000","last_active":"2024-03-26T21:29:55.889543+0000","last_peered":"2024-03-26T21:29:55.889543+0000","last_clean":"2024-03-26T21:29:55.889543+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:29:55.889543+0000","last_undegraded":"2024-03-26T21:29:55.889543+0000","last_fullsized":"2024-03-26T21:29:55.889543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:36.249302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.249302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:39.117721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.695403+0000","last_change":"2024-03-26T21:29:41.695430+0000","last_active":"2024-03-26T21:29:41.695403+0000","last_peered":"2024-03-26T21:29:41.695403+0000","last_clean":"2024-03-26T21:29:41.695403+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:29:41.695403+0000","last_undegraded":"2024-03-26T21:29:41.695403+0000","last_fullsized":"2024-03-26T21:29:41.695403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.695322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.695322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.842596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891920+0000","last_change":"2024-03-26T21:29:55.054708+0000","last_active":"2024-03-26T21:29:55.891920+0000","last_peered":"2024-03-26T21:29:55.891920+0000","last_clean":"2024-03-26T21:29:55.891920+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:29:55.891920+0000","last_undegraded":"2024-03-26T21:29:55.891920+0000","last_fullsized":"2024-03-26T21:29:55.891920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.054601+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.383564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071241+0000","last_change":"2024-03-26T21:28:38.874086+0000","last_active":"2024-03-26T21:29:41.071241+0000","last_peered":"2024-03-26T21:29:41.071241+0000","last_clean":"2024-03-26T21:29:41.071241+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:29:41.071241+0000","last_undegraded":"2024-03-26T21:29:41.071241+0000","last_fullsized":"2024-03-26T21:29:41.071241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.551108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891287+0000","last_change":"2024-03-26T21:28:40.887390+0000","last_active":"2024-03-26T21:29:55.891287+0000","last_peered":"2024-03-26T21:29:55.891287+0000","last_clean":"2024-03-26T21:29:55.891287+0000","last_became_active":"2024-03-26T21:28:40.886965+0000","last_became_peered":"2024-03-26T21:28:40.886965+0000","last_unstale":"2024-03-26T21:29:55.891287+0000","last_undegraded":"2024-03-26T21:29:55.891287+0000","last_fullsized":"2024-03-26T21:29:55.891287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":304,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583887+0000","last_change":"2024-03-26T21:28:42.275445+0000","last_active":"2024-03-26T21:29:41.583887+0000","last_peered":"2024-03-26T21:29:41.583887+0000","last_clean":"2024-03-26T21:29:41.583887+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:29:41.583887+0000","last_undegraded":"2024-03-26T21:29:41.583887+0000","last_fullsized":"2024-03-26T21:29:41.583887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T21:28:42.275350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:28:42.275350+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:52.670384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775577+0000","last_change":"2024-03-26T21:29:38.690870+0000","last_active":"2024-03-26T21:29:40.775577+0000","last_peered":"2024-03-26T21:29:40.775577+0000","last_clean":"2024-03-26T21:29:40.775577+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:29:40.775577+0000","last_undegraded":"2024-03-26T21:29:40.775577+0000","last_fullsized":"2024-03-26T21:29:40.775577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.690779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.690779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.230074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"88'24","reported_seq":104,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.903129+0000","last_change":"2024-03-26T21:29:55.903181+0000","last_active":"2024-03-26T21:29:55.903129+0000","last_peered":"2024-03-26T21:29:55.903129+0000","last_clean":"2024-03-26T21:29:55.903129+0000","last_became_active":"2024-03-26T21:28:37.140774+0000","last_became_peered":"2024-03-26T21:28:37.140774+0000","last_unstale":"2024-03-26T21:29:55.903129+0000","last_undegraded":"2024-03-26T21:29:55.903129+0000","last_fullsized":"2024-03-26T21:29:55.903129+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:13.673385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584298+0000","last_change":"2024-03-26T21:28:38.868716+0000","last_active":"2024-03-26T21:29:41.584298+0000","last_peered":"2024-03-26T21:29:41.584298+0000","last_clean":"2024-03-26T21:29:41.584298+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:29:41.584298+0000","last_undegraded":"2024-03-26T21:29:41.584298+0000","last_fullsized":"2024-03-26T21:29:41.584298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.403849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891271+0000","last_change":"2024-03-26T21:28:40.887424+0000","last_active":"2024-03-26T21:29:55.891271+0000","last_peered":"2024-03-26T21:29:55.891271+0000","last_clean":"2024-03-26T21:29:55.891271+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:29:55.891271+0000","last_undegraded":"2024-03-26T21:29:55.891271+0000","last_fullsized":"2024-03-26T21:29:55.891271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.086375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891131+0000","last_change":"2024-03-26T21:28:34.843666+0000","last_active":"2024-03-26T21:29:55.891131+0000","last_peered":"2024-03-26T21:29:55.891131+0000","last_clean":"2024-03-26T21:29:55.891131+0000","last_became_active":"2024-03-26T21:28:34.843392+0000","last_became_peered":"2024-03-26T21:28:34.843392+0000","last_unstale":"2024-03-26T21:29:55.891131+0000","last_undegraded":"2024-03-26T21:29:55.891131+0000","last_fullsized":"2024-03-26T21:29:55.891131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:54.465645+0000","last_change":"2024-03-26T21:28:38.848730+0000","last_active":"2024-03-26T21:29:54.465645+0000","last_peered":"2024-03-26T21:29:54.465645+0000","last_clean":"2024-03-26T21:29:54.465645+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:29:54.465645+0000","last_undegraded":"2024-03-26T21:29:54.465645+0000","last_fullsized":"2024-03-26T21:29:54.465645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891745+0000","last_change":"2024-03-26T21:29:38.920799+0000","last_active":"2024-03-26T21:29:55.891745+0000","last_peered":"2024-03-26T21:29:55.891745+0000","last_clean":"2024-03-26T21:29:55.891745+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:29:55.891745+0000","last_undegraded":"2024-03-26T21:29:55.891745+0000","last_fullsized":"2024-03-26T21:29:55.891745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:38.920701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.920701+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.589043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891510+0000","last_change":"2024-03-26T21:28:38.870606+0000","last_active":"2024-03-26T21:29:55.891510+0000","last_peered":"2024-03-26T21:29:55.891510+0000","last_clean":"2024-03-26T21:29:55.891510+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:29:55.891510+0000","last_undegraded":"2024-03-26T21:29:55.891510+0000","last_fullsized":"2024-03-26T21:29:55.891510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:07.288276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889655+0000","last_change":"2024-03-26T21:28:41.116399+0000","last_active":"2024-03-26T21:29:55.889655+0000","last_peered":"2024-03-26T21:29:55.889655+0000","last_clean":"2024-03-26T21:29:55.889655+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:29:55.889655+0000","last_undegraded":"2024-03-26T21:29:55.889655+0000","last_fullsized":"2024-03-26T21:29:55.889655+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.016224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891518+0000","last_change":"2024-03-26T21:29:42.167843+0000","last_active":"2024-03-26T21:29:55.891518+0000","last_peered":"2024-03-26T21:29:55.891518+0000","last_clean":"2024-03-26T21:29:55.891518+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:29:55.891518+0000","last_undegraded":"2024-03-26T21:29:55.891518+0000","last_fullsized":"2024-03-26T21:29:55.891518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.167719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:45.815629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":95,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891363+0000","last_change":"2024-03-26T21:29:51.863137+0000","last_active":"2024-03-26T21:29:55.891363+0000","last_peered":"2024-03-26T21:29:55.891363+0000","last_clean":"2024-03-26T21:29:55.891363+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:55.891363+0000","last_undegraded":"2024-03-26T21:29:55.891363+0000","last_fullsized":"2024-03-26T21:29:55.891363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:51.862960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.862960+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:12.041689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:52.956752+0000","last_change":"2024-03-26T21:29:52.956781+0000","last_active":"2024-03-26T21:29:52.956752+0000","last_peered":"2024-03-26T21:29:52.956752+0000","last_clean":"2024-03-26T21:29:52.956752+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:29:52.956752+0000","last_undegraded":"2024-03-26T21:29:52.956752+0000","last_fullsized":"2024-03-26T21:29:52.956752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.956669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.956669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:54.531858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889092+0000","last_change":"2024-03-26T21:28:40.890739+0000","last_active":"2024-03-26T21:29:55.889092+0000","last_peered":"2024-03-26T21:29:55.889092+0000","last_clean":"2024-03-26T21:29:55.889092+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:29:55.889092+0000","last_undegraded":"2024-03-26T21:29:55.889092+0000","last_fullsized":"2024-03-26T21:29:55.889092+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890929+0000","last_change":"2024-03-26T21:29:38.189988+0000","last_active":"2024-03-26T21:29:55.890929+0000","last_peered":"2024-03-26T21:29:55.890929+0000","last_clean":"2024-03-26T21:29:55.890929+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:29:55.890929+0000","last_undegraded":"2024-03-26T21:29:55.890929+0000","last_fullsized":"2024-03-26T21:29:55.890929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.189877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.189877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:38.248593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890438+0000","last_change":"2024-03-26T21:28:36.844629+0000","last_active":"2024-03-26T21:29:55.890438+0000","last_peered":"2024-03-26T21:29:55.890438+0000","last_clean":"2024-03-26T21:29:55.890438+0000","last_became_active":"2024-03-26T21:28:36.841759+0000","last_became_peered":"2024-03-26T21:28:36.841759+0000","last_unstale":"2024-03-26T21:29:55.890438+0000","last_undegraded":"2024-03-26T21:29:55.890438+0000","last_fullsized":"2024-03-26T21:29:55.890438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891123+0000","last_change":"2024-03-26T21:28:38.869750+0000","last_active":"2024-03-26T21:29:55.891123+0000","last_peered":"2024-03-26T21:29:55.891123+0000","last_clean":"2024-03-26T21:29:55.891123+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:29:55.891123+0000","last_undegraded":"2024-03-26T21:29:55.891123+0000","last_fullsized":"2024-03-26T21:29:55.891123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584148+0000","last_change":"2024-03-26T21:28:41.115853+0000","last_active":"2024-03-26T21:29:41.584148+0000","last_peered":"2024-03-26T21:29:41.584148+0000","last_clean":"2024-03-26T21:29:41.584148+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:29:41.584148+0000","last_undegraded":"2024-03-26T21:29:41.584148+0000","last_fullsized":"2024-03-26T21:29:41.584148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.333646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891860+0000","last_change":"2024-03-26T21:28:34.846498+0000","last_active":"2024-03-26T21:29:55.891860+0000","last_peered":"2024-03-26T21:29:55.891860+0000","last_clean":"2024-03-26T21:29:55.891860+0000","last_became_active":"2024-03-26T21:28:34.846219+0000","last_became_peered":"2024-03-26T21:28:34.846219+0000","last_unstale":"2024-03-26T21:29:55.891860+0000","last_undegraded":"2024-03-26T21:29:55.891860+0000","last_fullsized":"2024-03-26T21:29:55.891860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771159+0000","last_change":"2024-03-26T21:28:36.840120+0000","last_active":"2024-03-26T21:29:40.771159+0000","last_peered":"2024-03-26T21:29:40.771159+0000","last_clean":"2024-03-26T21:29:40.771159+0000","last_became_active":"2024-03-26T21:28:36.839780+0000","last_became_peered":"2024-03-26T21:28:36.839780+0000","last_unstale":"2024-03-26T21:29:40.771159+0000","last_undegraded":"2024-03-26T21:29:40.771159+0000","last_fullsized":"2024-03-26T21:29:40.771159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.000016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899619+0000","last_change":"2024-03-26T21:28:38.864688+0000","last_active":"2024-03-26T21:29:49.899619+0000","last_peered":"2024-03-26T21:29:49.899619+0000","last_clean":"2024-03-26T21:29:49.899619+0000","last_became_active":"2024-03-26T21:28:38.864364+0000","last_became_peered":"2024-03-26T21:28:38.864364+0000","last_unstale":"2024-03-26T21:29:49.899619+0000","last_undegraded":"2024-03-26T21:29:49.899619+0000","last_fullsized":"2024-03-26T21:29:49.899619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.632429+0000","last_change":"2024-03-26T21:29:48.632454+0000","last_active":"2024-03-26T21:29:48.632429+0000","last_peered":"2024-03-26T21:29:48.632429+0000","last_clean":"2024-03-26T21:29:48.632429+0000","last_became_active":"2024-03-26T21:28:40.886497+0000","last_became_peered":"2024-03-26T21:28:40.886497+0000","last_unstale":"2024-03-26T21:29:48.632429+0000","last_undegraded":"2024-03-26T21:29:48.632429+0000","last_fullsized":"2024-03-26T21:29:48.632429+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.661591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776254+0000","last_change":"2024-03-26T21:28:34.839825+0000","last_active":"2024-03-26T21:29:40.776254+0000","last_peered":"2024-03-26T21:29:40.776254+0000","last_clean":"2024-03-26T21:29:40.776254+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:29:40.776254+0000","last_undegraded":"2024-03-26T21:29:40.776254+0000","last_fullsized":"2024-03-26T21:29:40.776254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:41.674843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775290+0000","last_change":"2024-03-26T21:28:37.143706+0000","last_active":"2024-03-26T21:29:40.775290+0000","last_peered":"2024-03-26T21:29:40.775290+0000","last_clean":"2024-03-26T21:29:40.775290+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:29:40.775290+0000","last_undegraded":"2024-03-26T21:29:40.775290+0000","last_fullsized":"2024-03-26T21:29:40.775290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.770981+0000","last_change":"2024-03-26T21:28:38.863832+0000","last_active":"2024-03-26T21:29:40.770981+0000","last_peered":"2024-03-26T21:29:40.770981+0000","last_clean":"2024-03-26T21:29:40.770981+0000","last_became_active":"2024-03-26T21:28:38.863503+0000","last_became_peered":"2024-03-26T21:28:38.863503+0000","last_unstale":"2024-03-26T21:29:40.770981+0000","last_undegraded":"2024-03-26T21:29:40.770981+0000","last_fullsized":"2024-03-26T21:29:40.770981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.090750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890764+0000","last_change":"2024-03-26T21:28:40.891483+0000","last_active":"2024-03-26T21:29:55.890764+0000","last_peered":"2024-03-26T21:29:55.890764+0000","last_clean":"2024-03-26T21:29:55.890764+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:29:55.890764+0000","last_undegraded":"2024-03-26T21:29:55.890764+0000","last_fullsized":"2024-03-26T21:29:55.890764+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899872+0000","last_change":"2024-03-26T21:28:38.871742+0000","last_active":"2024-03-26T21:29:49.899872+0000","last_peered":"2024-03-26T21:29:49.899872+0000","last_clean":"2024-03-26T21:29:49.899872+0000","last_became_active":"2024-03-26T21:28:38.871425+0000","last_became_peered":"2024-03-26T21:28:38.871425+0000","last_unstale":"2024-03-26T21:29:49.899872+0000","last_undegraded":"2024-03-26T21:29:49.899872+0000","last_fullsized":"2024-03-26T21:29:49.899872+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.473329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.689412+0000","last_change":"2024-03-26T21:29:41.689438+0000","last_active":"2024-03-26T21:29:41.689412+0000","last_peered":"2024-03-26T21:29:41.689412+0000","last_clean":"2024-03-26T21:29:41.689412+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:29:41.689412+0000","last_undegraded":"2024-03-26T21:29:41.689412+0000","last_fullsized":"2024-03-26T21:29:41.689412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.689332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.689332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.552879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.892132+0000","last_change":"2024-03-26T21:28:36.848001+0000","last_active":"2024-03-26T21:29:55.892132+0000","last_peered":"2024-03-26T21:29:55.892132+0000","last_clean":"2024-03-26T21:29:55.892132+0000","last_became_active":"2024-03-26T21:28:36.847705+0000","last_became_peered":"2024-03-26T21:28:36.847705+0000","last_unstale":"2024-03-26T21:29:55.892132+0000","last_undegraded":"2024-03-26T21:29:55.892132+0000","last_fullsized":"2024-03-26T21:29:55.892132+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:03.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775503+0000","last_change":"2024-03-26T21:28:41.113828+0000","last_active":"2024-03-26T21:29:40.775503+0000","last_peered":"2024-03-26T21:29:40.775503+0000","last_clean":"2024-03-26T21:29:40.775503+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:29:40.775503+0000","last_undegraded":"2024-03-26T21:29:40.775503+0000","last_fullsized":"2024-03-26T21:29:40.775503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:51.030485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891464+0000","last_change":"2024-03-26T21:28:38.872061+0000","last_active":"2024-03-26T21:29:55.891464+0000","last_peered":"2024-03-26T21:29:55.891464+0000","last_clean":"2024-03-26T21:29:55.891464+0000","last_became_active":"2024-03-26T21:28:38.869931+0000","last_became_peered":"2024-03-26T21:28:38.869931+0000","last_unstale":"2024-03-26T21:29:55.891464+0000","last_undegraded":"2024-03-26T21:29:55.891464+0000","last_fullsized":"2024-03-26T21:29:55.891464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.256868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:50.575688+0000","last_change":"2024-03-26T21:29:50.575715+0000","last_active":"2024-03-26T21:29:50.575688+0000","last_peered":"2024-03-26T21:29:50.575688+0000","last_clean":"2024-03-26T21:29:50.575688+0000","last_became_active":"2024-03-26T21:28:34.842398+0000","last_became_peered":"2024-03-26T21:28:34.842398+0000","last_unstale":"2024-03-26T21:29:50.575688+0000","last_undegraded":"2024-03-26T21:29:50.575688+0000","last_fullsized":"2024-03-26T21:29:50.575688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.357015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771980+0000","last_change":"2024-03-26T21:28:36.842091+0000","last_active":"2024-03-26T21:29:40.771980+0000","last_peered":"2024-03-26T21:29:40.771980+0000","last_clean":"2024-03-26T21:29:40.771980+0000","last_became_active":"2024-03-26T21:28:36.841703+0000","last_became_peered":"2024-03-26T21:28:36.841703+0000","last_unstale":"2024-03-26T21:29:40.771980+0000","last_undegraded":"2024-03-26T21:29:40.771980+0000","last_fullsized":"2024-03-26T21:29:40.771980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:51.833347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071374+0000","last_change":"2024-03-26T21:28:40.885118+0000","last_active":"2024-03-26T21:29:41.071374+0000","last_peered":"2024-03-26T21:29:41.071374+0000","last_clean":"2024-03-26T21:29:41.071374+0000","last_became_active":"2024-03-26T21:28:40.884823+0000","last_became_peered":"2024-03-26T21:28:40.884823+0000","last_unstale":"2024-03-26T21:29:41.071374+0000","last_undegraded":"2024-03-26T21:29:41.071374+0000","last_fullsized":"2024-03-26T21:29:41.071374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:59.320539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891719+0000","last_change":"2024-03-26T21:29:40.185232+0000","last_active":"2024-03-26T21:29:55.891719+0000","last_peered":"2024-03-26T21:29:55.891719+0000","last_clean":"2024-03-26T21:29:55.891719+0000","last_became_active":"2024-03-26T21:28:38.867110+0000","last_became_peered":"2024-03-26T21:28:38.867110+0000","last_unstale":"2024-03-26T21:29:55.891719+0000","last_undegraded":"2024-03-26T21:29:55.891719+0000","last_fullsized":"2024-03-26T21:29:55.891719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.295284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.864594+0000","last_change":"2024-03-26T21:29:39.867915+0000","last_active":"2024-03-26T21:29:40.864594+0000","last_peered":"2024-03-26T21:29:40.864594+0000","last_clean":"2024-03-26T21:29:40.864594+0000","last_became_active":"2024-03-26T21:28:34.846385+0000","last_became_peered":"2024-03-26T21:28:34.846385+0000","last_unstale":"2024-03-26T21:29:40.864594+0000","last_undegraded":"2024-03-26T21:29:40.864594+0000","last_fullsized":"2024-03-26T21:29:40.864594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:51.580336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771735+0000","last_change":"2024-03-26T21:28:36.842153+0000","last_active":"2024-03-26T21:29:40.771735+0000","last_peered":"2024-03-26T21:29:40.771735+0000","last_clean":"2024-03-26T21:29:40.771735+0000","last_became_active":"2024-03-26T21:28:36.841769+0000","last_became_peered":"2024-03-26T21:28:36.841769+0000","last_unstale":"2024-03-26T21:29:40.771735+0000","last_undegraded":"2024-03-26T21:29:40.771735+0000","last_fullsized":"2024-03-26T21:29:40.771735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.653845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772133+0000","last_change":"2024-03-26T21:28:40.880305+0000","last_active":"2024-03-26T21:29:40.772133+0000","last_peered":"2024-03-26T21:29:40.772133+0000","last_clean":"2024-03-26T21:29:40.772133+0000","last_became_active":"2024-03-26T21:28:40.879974+0000","last_became_peered":"2024-03-26T21:28:40.879974+0000","last_unstale":"2024-03-26T21:29:40.772133+0000","last_undegraded":"2024-03-26T21:29:40.772133+0000","last_fullsized":"2024-03-26T21:29:40.772133+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899652+0000","last_change":"2024-03-26T21:28:38.867661+0000","last_active":"2024-03-26T21:29:49.899652+0000","last_peered":"2024-03-26T21:29:49.899652+0000","last_clean":"2024-03-26T21:29:49.899652+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:29:49.899652+0000","last_undegraded":"2024-03-26T21:29:49.899652+0000","last_fullsized":"2024-03-26T21:29:49.899652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776015+0000","last_change":"2024-03-26T21:28:34.848313+0000","last_active":"2024-03-26T21:29:40.776015+0000","last_peered":"2024-03-26T21:29:40.776015+0000","last_clean":"2024-03-26T21:29:40.776015+0000","last_became_active":"2024-03-26T21:28:34.848009+0000","last_became_peered":"2024-03-26T21:28:34.848009+0000","last_unstale":"2024-03-26T21:29:40.776015+0000","last_undegraded":"2024-03-26T21:29:40.776015+0000","last_fullsized":"2024-03-26T21:29:40.776015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.325454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771552+0000","last_change":"2024-03-26T21:29:37.647253+0000","last_active":"2024-03-26T21:29:40.771552+0000","last_peered":"2024-03-26T21:29:40.771552+0000","last_clean":"2024-03-26T21:29:40.771552+0000","last_became_active":"2024-03-26T21:28:36.847902+0000","last_became_peered":"2024-03-26T21:28:36.847902+0000","last_unstale":"2024-03-26T21:29:40.771552+0000","last_undegraded":"2024-03-26T21:29:40.771552+0000","last_fullsized":"2024-03-26T21:29:40.771552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.419033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1392640,"data_stored":937448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364827,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28344,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716808,"statfs":{"total":95995035648,"available":95966011392,"internally_reserved":0,"allocated":1359872,"data_stored":935036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":353090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":936754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":761856,"data_stored":344783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":345572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":346620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":786432,"data_stored":345352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:30:00.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:30:01.130 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:00 smithi067 ceph-mon[33346]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:00 smithi067 ceph-mon[33346]: overall HEALTH_OK 2024-03-26T21:30:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:00 smithi067 ceph-mon[33346]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:00 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[29406]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[29406]: overall HEALTH_OK 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[29406]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[40587]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[40587]: overall HEALTH_OK 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[40587]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:01.252 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:30:02.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:30:02.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 3.19 deep-scrub starts 2024-03-26T21:30:02.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 3.19 deep-scrub ok 2024-03-26T21:30:02.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:30:02.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:01 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:30:02.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 3.19 deep-scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 3.19 deep-scrub ok 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 3.19 deep-scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 3.19 deep-scrub ok 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:30:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:01 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:30:03.145 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:30:03.147 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:30:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:30:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:30:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:30:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:30:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:30:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:30:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:30:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:02 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:30:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:30:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:30:03.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:02 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:30:04.048 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-26T21:30:01.279953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1009,"num_read_kb":855,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240740,"kb_used_data":7968,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720476,"statfs":{"total":767960285184,"available":767713767424,"internally_reserved":0,"allocated":8159232,"data_stored":4557673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001984"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771079+0000","last_change":"2024-03-26T21:28:40.880308+0000","last_active":"2024-03-26T21:29:40.771079+0000","last_peered":"2024-03-26T21:29:40.771079+0000","last_clean":"2024-03-26T21:29:40.771079+0000","last_became_active":"2024-03-26T21:28:40.879971+0000","last_became_peered":"2024-03-26T21:28:40.879971+0000","last_unstale":"2024-03-26T21:29:40.771079+0000","last_undegraded":"2024-03-26T21:29:40.771079+0000","last_fullsized":"2024-03-26T21:29:40.771079+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:52.156812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889756+0000","last_change":"2024-03-26T21:28:38.866679+0000","last_active":"2024-03-26T21:29:55.889756+0000","last_peered":"2024-03-26T21:29:55.889756+0000","last_clean":"2024-03-26T21:29:55.889756+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:29:55.889756+0000","last_undegraded":"2024-03-26T21:29:55.889756+0000","last_fullsized":"2024-03-26T21:29:55.889756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:03.904348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890820+0000","last_change":"2024-03-26T21:28:34.844384+0000","last_active":"2024-03-26T21:29:55.890820+0000","last_peered":"2024-03-26T21:29:55.890820+0000","last_clean":"2024-03-26T21:29:55.890820+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:29:55.890820+0000","last_undegraded":"2024-03-26T21:29:55.890820+0000","last_fullsized":"2024-03-26T21:29:55.890820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:51.544689+0000","last_change":"2024-03-26T21:29:51.544731+0000","last_active":"2024-03-26T21:29:51.544689+0000","last_peered":"2024-03-26T21:29:51.544689+0000","last_clean":"2024-03-26T21:29:51.544689+0000","last_became_active":"2024-03-26T21:28:36.841677+0000","last_became_peered":"2024-03-26T21:28:36.841677+0000","last_unstale":"2024-03-26T21:29:51.544689+0000","last_undegraded":"2024-03-26T21:29:51.544689+0000","last_fullsized":"2024-03-26T21:29:51.544689+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.477780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.749455+0000","last_change":"2024-03-26T21:29:49.749475+0000","last_active":"2024-03-26T21:29:49.749455+0000","last_peered":"2024-03-26T21:29:49.749455+0000","last_clean":"2024-03-26T21:29:49.749455+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:29:49.749455+0000","last_undegraded":"2024-03-26T21:29:49.749455+0000","last_fullsized":"2024-03-26T21:29:49.749455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:49.749386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:49.749386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.347098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:45.645645+0000","last_change":"2024-03-26T21:29:45.645672+0000","last_active":"2024-03-26T21:29:45.645645+0000","last_peered":"2024-03-26T21:29:45.645645+0000","last_clean":"2024-03-26T21:29:45.645645+0000","last_became_active":"2024-03-26T21:28:38.866501+0000","last_became_peered":"2024-03-26T21:28:38.866501+0000","last_unstale":"2024-03-26T21:29:45.645645+0000","last_undegraded":"2024-03-26T21:29:45.645645+0000","last_fullsized":"2024-03-26T21:29:45.645645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.799670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:50.735259+0000","last_change":"2024-03-26T21:29:50.735279+0000","last_active":"2024-03-26T21:29:50.735259+0000","last_peered":"2024-03-26T21:29:50.735259+0000","last_clean":"2024-03-26T21:29:50.735259+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:29:50.735259+0000","last_undegraded":"2024-03-26T21:29:50.735259+0000","last_fullsized":"2024-03-26T21:29:50.735259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.735199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.735199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.656417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775649+0000","last_change":"2024-03-26T21:28:36.846364+0000","last_active":"2024-03-26T21:29:40.775649+0000","last_peered":"2024-03-26T21:29:40.775649+0000","last_clean":"2024-03-26T21:29:40.775649+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:29:40.775649+0000","last_undegraded":"2024-03-26T21:29:40.775649+0000","last_fullsized":"2024-03-26T21:29:40.775649+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:47.364815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889359+0000","last_change":"2024-03-26T21:28:40.888299+0000","last_active":"2024-03-26T21:29:55.889359+0000","last_peered":"2024-03-26T21:29:55.889359+0000","last_clean":"2024-03-26T21:29:55.889359+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:29:55.889359+0000","last_undegraded":"2024-03-26T21:29:55.889359+0000","last_fullsized":"2024-03-26T21:29:55.889359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890581+0000","last_change":"2024-03-26T21:28:38.872667+0000","last_active":"2024-03-26T21:29:55.890581+0000","last_peered":"2024-03-26T21:29:55.890581+0000","last_clean":"2024-03-26T21:29:55.890581+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:29:55.890581+0000","last_undegraded":"2024-03-26T21:29:55.890581+0000","last_fullsized":"2024-03-26T21:29:55.890581+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.510486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890783+0000","last_change":"2024-03-26T21:28:34.842302+0000","last_active":"2024-03-26T21:29:55.890783+0000","last_peered":"2024-03-26T21:29:55.890783+0000","last_clean":"2024-03-26T21:29:55.890783+0000","last_became_active":"2024-03-26T21:28:34.841544+0000","last_became_peered":"2024-03-26T21:28:34.841544+0000","last_unstale":"2024-03-26T21:29:55.890783+0000","last_undegraded":"2024-03-26T21:29:55.890783+0000","last_fullsized":"2024-03-26T21:29:55.890783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.708588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:52.496798+0000","last_change":"2024-03-26T21:29:52.496860+0000","last_active":"2024-03-26T21:29:52.496798+0000","last_peered":"2024-03-26T21:29:52.496798+0000","last_clean":"2024-03-26T21:29:52.496798+0000","last_became_active":"2024-03-26T21:28:36.847804+0000","last_became_peered":"2024-03-26T21:28:36.847804+0000","last_unstale":"2024-03-26T21:29:52.496798+0000","last_undegraded":"2024-03-26T21:29:52.496798+0000","last_fullsized":"2024-03-26T21:29:52.496798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.032728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890297+0000","last_change":"2024-03-26T21:28:40.887065+0000","last_active":"2024-03-26T21:29:55.890297+0000","last_peered":"2024-03-26T21:29:55.890297+0000","last_clean":"2024-03-26T21:29:55.890297+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:29:55.890297+0000","last_undegraded":"2024-03-26T21:29:55.890297+0000","last_fullsized":"2024-03-26T21:29:55.890297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583919+0000","last_change":"2024-03-26T21:28:38.864721+0000","last_active":"2024-03-26T21:29:41.583919+0000","last_peered":"2024-03-26T21:29:41.583919+0000","last_clean":"2024-03-26T21:29:41.583919+0000","last_became_active":"2024-03-26T21:28:38.864406+0000","last_became_peered":"2024-03-26T21:28:38.864406+0000","last_unstale":"2024-03-26T21:29:41.583919+0000","last_undegraded":"2024-03-26T21:29:41.583919+0000","last_fullsized":"2024-03-26T21:29:41.583919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:43.916601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890719+0000","last_change":"2024-03-26T21:28:34.847255+0000","last_active":"2024-03-26T21:29:55.890719+0000","last_peered":"2024-03-26T21:29:55.890719+0000","last_clean":"2024-03-26T21:29:55.890719+0000","last_became_active":"2024-03-26T21:28:34.846956+0000","last_became_peered":"2024-03-26T21:28:34.846956+0000","last_unstale":"2024-03-26T21:29:55.890719+0000","last_undegraded":"2024-03-26T21:29:55.890719+0000","last_fullsized":"2024-03-26T21:29:55.890719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.186750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771559+0000","last_change":"2024-03-26T21:28:36.846687+0000","last_active":"2024-03-26T21:29:40.771559+0000","last_peered":"2024-03-26T21:29:40.771559+0000","last_clean":"2024-03-26T21:29:40.771559+0000","last_became_active":"2024-03-26T21:28:36.846364+0000","last_became_peered":"2024-03-26T21:28:36.846364+0000","last_unstale":"2024-03-26T21:29:40.771559+0000","last_undegraded":"2024-03-26T21:29:40.771559+0000","last_fullsized":"2024-03-26T21:29:40.771559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.374326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772076+0000","last_change":"2024-03-26T21:28:41.114040+0000","last_active":"2024-03-26T21:29:40.772076+0000","last_peered":"2024-03-26T21:29:40.772076+0000","last_clean":"2024-03-26T21:29:40.772076+0000","last_became_active":"2024-03-26T21:28:41.113734+0000","last_became_peered":"2024-03-26T21:28:41.113734+0000","last_unstale":"2024-03-26T21:29:40.772076+0000","last_undegraded":"2024-03-26T21:29:40.772076+0000","last_fullsized":"2024-03-26T21:29:40.772076+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.261763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891379+0000","last_change":"2024-03-26T21:28:38.871499+0000","last_active":"2024-03-26T21:29:55.891379+0000","last_peered":"2024-03-26T21:29:55.891379+0000","last_clean":"2024-03-26T21:29:55.891379+0000","last_became_active":"2024-03-26T21:28:38.870984+0000","last_became_peered":"2024-03-26T21:28:38.870984+0000","last_unstale":"2024-03-26T21:29:55.891379+0000","last_undegraded":"2024-03-26T21:29:55.891379+0000","last_fullsized":"2024-03-26T21:29:55.891379+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891278+0000","last_change":"2024-03-26T21:28:34.847658+0000","last_active":"2024-03-26T21:29:55.891278+0000","last_peered":"2024-03-26T21:29:55.891278+0000","last_clean":"2024-03-26T21:29:55.891278+0000","last_became_active":"2024-03-26T21:28:34.846913+0000","last_became_peered":"2024-03-26T21:28:34.846913+0000","last_unstale":"2024-03-26T21:29:55.891278+0000","last_undegraded":"2024-03-26T21:29:55.891278+0000","last_fullsized":"2024-03-26T21:29:55.891278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:45.654328+0000","last_change":"2024-03-26T21:29:45.654379+0000","last_active":"2024-03-26T21:29:45.654328+0000","last_peered":"2024-03-26T21:29:45.654328+0000","last_clean":"2024-03-26T21:29:45.654328+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:29:45.654328+0000","last_undegraded":"2024-03-26T21:29:45.654328+0000","last_fullsized":"2024-03-26T21:29:45.654328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:29:45.654238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.654238+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.970402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776109+0000","last_change":"2024-03-26T21:28:41.113863+0000","last_active":"2024-03-26T21:29:40.776109+0000","last_peered":"2024-03-26T21:29:40.776109+0000","last_clean":"2024-03-26T21:29:40.776109+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:29:40.776109+0000","last_undegraded":"2024-03-26T21:29:40.776109+0000","last_fullsized":"2024-03-26T21:29:40.776109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.033803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:44.615977+0000","last_change":"2024-03-26T21:29:44.616004+0000","last_active":"2024-03-26T21:29:44.615977+0000","last_peered":"2024-03-26T21:29:44.615977+0000","last_clean":"2024-03-26T21:29:44.615977+0000","last_became_active":"2024-03-26T21:28:38.867256+0000","last_became_peered":"2024-03-26T21:28:38.867256+0000","last_unstale":"2024-03-26T21:29:44.615977+0000","last_undegraded":"2024-03-26T21:29:44.615977+0000","last_fullsized":"2024-03-26T21:29:44.615977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.353334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771290+0000","last_change":"2024-03-26T21:28:34.846901+0000","last_active":"2024-03-26T21:29:40.771290+0000","last_peered":"2024-03-26T21:29:40.771290+0000","last_clean":"2024-03-26T21:29:40.771290+0000","last_became_active":"2024-03-26T21:28:34.846583+0000","last_became_peered":"2024-03-26T21:28:34.846583+0000","last_unstale":"2024-03-26T21:29:40.771290+0000","last_undegraded":"2024-03-26T21:29:40.771290+0000","last_fullsized":"2024-03-26T21:29:40.771290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:53.292403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775827+0000","last_change":"2024-03-26T21:28:36.850322+0000","last_active":"2024-03-26T21:29:40.775827+0000","last_peered":"2024-03-26T21:29:40.775827+0000","last_clean":"2024-03-26T21:29:40.775827+0000","last_became_active":"2024-03-26T21:28:36.849968+0000","last_became_peered":"2024-03-26T21:28:36.849968+0000","last_unstale":"2024-03-26T21:29:40.775827+0000","last_undegraded":"2024-03-26T21:29:40.775827+0000","last_fullsized":"2024-03-26T21:29:40.775827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:54.614142+0000","last_change":"2024-03-26T21:29:54.614182+0000","last_active":"2024-03-26T21:29:54.614142+0000","last_peered":"2024-03-26T21:29:54.614142+0000","last_clean":"2024-03-26T21:29:54.614142+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:29:54.614142+0000","last_undegraded":"2024-03-26T21:29:54.614142+0000","last_fullsized":"2024-03-26T21:29:54.614142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:29:54.614064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:54.614064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:06.934031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.861072+0000","last_change":"2024-03-26T21:28:38.867627+0000","last_active":"2024-03-26T21:29:56.861072+0000","last_peered":"2024-03-26T21:29:56.861072+0000","last_clean":"2024-03-26T21:29:56.861072+0000","last_became_active":"2024-03-26T21:28:38.867245+0000","last_became_peered":"2024-03-26T21:28:38.867245+0000","last_unstale":"2024-03-26T21:29:56.861072+0000","last_undegraded":"2024-03-26T21:29:56.861072+0000","last_fullsized":"2024-03-26T21:29:56.861072+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.894707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889851+0000","last_change":"2024-03-26T21:28:34.846724+0000","last_active":"2024-03-26T21:29:55.889851+0000","last_peered":"2024-03-26T21:29:55.889851+0000","last_clean":"2024-03-26T21:29:55.889851+0000","last_became_active":"2024-03-26T21:28:34.846408+0000","last_became_peered":"2024-03-26T21:28:34.846408+0000","last_unstale":"2024-03-26T21:29:55.889851+0000","last_undegraded":"2024-03-26T21:29:55.889851+0000","last_fullsized":"2024-03-26T21:29:55.889851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891869+0000","last_change":"2024-03-26T21:28:37.143422+0000","last_active":"2024-03-26T21:29:55.891869+0000","last_peered":"2024-03-26T21:29:55.891869+0000","last_clean":"2024-03-26T21:29:55.891869+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:29:55.891869+0000","last_undegraded":"2024-03-26T21:29:55.891869+0000","last_fullsized":"2024-03-26T21:29:55.891869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.464940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891044+0000","last_change":"2024-03-26T21:29:45.092723+0000","last_active":"2024-03-26T21:29:55.891044+0000","last_peered":"2024-03-26T21:29:55.891044+0000","last_clean":"2024-03-26T21:29:55.891044+0000","last_became_active":"2024-03-26T21:28:40.881781+0000","last_became_peered":"2024-03-26T21:28:40.881781+0000","last_unstale":"2024-03-26T21:29:55.891044+0000","last_undegraded":"2024-03-26T21:29:55.891044+0000","last_fullsized":"2024-03-26T21:29:55.891044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.160967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.892051+0000","last_change":"2024-03-26T21:28:38.867746+0000","last_active":"2024-03-26T21:29:55.892051+0000","last_peered":"2024-03-26T21:29:55.892051+0000","last_clean":"2024-03-26T21:29:55.892051+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:29:55.892051+0000","last_undegraded":"2024-03-26T21:29:55.892051+0000","last_fullsized":"2024-03-26T21:29:55.892051+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775121+0000","last_change":"2024-03-26T21:28:34.849241+0000","last_active":"2024-03-26T21:29:40.775121+0000","last_peered":"2024-03-26T21:29:40.775121+0000","last_clean":"2024-03-26T21:29:40.775121+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:29:40.775121+0000","last_undegraded":"2024-03-26T21:29:40.775121+0000","last_fullsized":"2024-03-26T21:29:40.775121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:45.919934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.859931+0000","last_change":"2024-03-26T21:29:36.821598+0000","last_active":"2024-03-26T21:29:56.859931+0000","last_peered":"2024-03-26T21:29:56.859931+0000","last_clean":"2024-03-26T21:29:56.859931+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:29:56.859931+0000","last_undegraded":"2024-03-26T21:29:56.859931+0000","last_fullsized":"2024-03-26T21:29:56.859931+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:29:36.821436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.821436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.432234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583914+0000","last_change":"2024-03-26T21:28:40.885889+0000","last_active":"2024-03-26T21:29:41.583914+0000","last_peered":"2024-03-26T21:29:41.583914+0000","last_clean":"2024-03-26T21:29:41.583914+0000","last_became_active":"2024-03-26T21:28:40.885572+0000","last_became_peered":"2024-03-26T21:28:40.885572+0000","last_unstale":"2024-03-26T21:29:41.583914+0000","last_undegraded":"2024-03-26T21:29:41.583914+0000","last_fullsized":"2024-03-26T21:29:41.583914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:44.002943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.860175+0000","last_change":"2024-03-26T21:28:38.870467+0000","last_active":"2024-03-26T21:29:56.860175+0000","last_peered":"2024-03-26T21:29:56.860175+0000","last_clean":"2024-03-26T21:29:56.860175+0000","last_became_active":"2024-03-26T21:28:38.870142+0000","last_became_peered":"2024-03-26T21:28:38.870142+0000","last_unstale":"2024-03-26T21:29:56.860175+0000","last_undegraded":"2024-03-26T21:29:56.860175+0000","last_fullsized":"2024-03-26T21:29:56.860175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890725+0000","last_change":"2024-03-26T21:29:52.893256+0000","last_active":"2024-03-26T21:29:55.890725+0000","last_peered":"2024-03-26T21:29:55.890725+0000","last_clean":"2024-03-26T21:29:55.890725+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:29:55.890725+0000","last_undegraded":"2024-03-26T21:29:55.890725+0000","last_fullsized":"2024-03-26T21:29:55.890725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.893078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.893078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.394353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775121+0000","last_change":"2024-03-26T21:28:36.840820+0000","last_active":"2024-03-26T21:29:40.775121+0000","last_peered":"2024-03-26T21:29:40.775121+0000","last_clean":"2024-03-26T21:29:40.775121+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:29:40.775121+0000","last_undegraded":"2024-03-26T21:29:40.775121+0000","last_fullsized":"2024-03-26T21:29:40.775121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:44.639728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775387+0000","last_change":"2024-03-26T21:28:41.117667+0000","last_active":"2024-03-26T21:29:40.775387+0000","last_peered":"2024-03-26T21:29:40.775387+0000","last_clean":"2024-03-26T21:29:40.775387+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:29:40.775387+0000","last_undegraded":"2024-03-26T21:29:40.775387+0000","last_fullsized":"2024-03-26T21:29:40.775387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.963275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584112+0000","last_change":"2024-03-26T21:28:38.864935+0000","last_active":"2024-03-26T21:29:41.584112+0000","last_peered":"2024-03-26T21:29:41.584112+0000","last_clean":"2024-03-26T21:29:41.584112+0000","last_became_active":"2024-03-26T21:28:38.864421+0000","last_became_peered":"2024-03-26T21:28:38.864421+0000","last_unstale":"2024-03-26T21:29:41.584112+0000","last_undegraded":"2024-03-26T21:29:41.584112+0000","last_fullsized":"2024-03-26T21:29:41.584112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:53.514362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891131+0000","last_change":"2024-03-26T21:28:34.847955+0000","last_active":"2024-03-26T21:29:55.891131+0000","last_peered":"2024-03-26T21:29:55.891131+0000","last_clean":"2024-03-26T21:29:55.891131+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:29:55.891131+0000","last_undegraded":"2024-03-26T21:29:55.891131+0000","last_fullsized":"2024-03-26T21:29:55.891131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775545+0000","last_change":"2024-03-26T21:28:37.142935+0000","last_active":"2024-03-26T21:29:40.775545+0000","last_peered":"2024-03-26T21:29:40.775545+0000","last_clean":"2024-03-26T21:29:40.775545+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:29:40.775545+0000","last_undegraded":"2024-03-26T21:29:40.775545+0000","last_fullsized":"2024-03-26T21:29:40.775545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.315521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889091+0000","last_change":"2024-03-26T21:28:40.889084+0000","last_active":"2024-03-26T21:29:55.889091+0000","last_peered":"2024-03-26T21:29:55.889091+0000","last_clean":"2024-03-26T21:29:55.889091+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:29:55.889091+0000","last_undegraded":"2024-03-26T21:29:55.889091+0000","last_fullsized":"2024-03-26T21:29:55.889091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.980105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776180+0000","last_change":"2024-03-26T21:28:38.865685+0000","last_active":"2024-03-26T21:29:40.776180+0000","last_peered":"2024-03-26T21:29:40.776180+0000","last_clean":"2024-03-26T21:29:40.776180+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:29:40.776180+0000","last_undegraded":"2024-03-26T21:29:40.776180+0000","last_fullsized":"2024-03-26T21:29:40.776180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:46.796618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.577570+0000","last_change":"2024-03-26T21:29:55.577602+0000","last_active":"2024-03-26T21:29:55.577570+0000","last_peered":"2024-03-26T21:29:55.577570+0000","last_clean":"2024-03-26T21:29:55.577570+0000","last_became_active":"2024-03-26T21:28:34.848518+0000","last_became_peered":"2024-03-26T21:28:34.848518+0000","last_unstale":"2024-03-26T21:29:55.577570+0000","last_undegraded":"2024-03-26T21:29:55.577570+0000","last_fullsized":"2024-03-26T21:29:55.577570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.145785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890534+0000","last_change":"2024-03-26T21:28:36.841238+0000","last_active":"2024-03-26T21:29:55.890534+0000","last_peered":"2024-03-26T21:29:55.890534+0000","last_clean":"2024-03-26T21:29:55.890534+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:29:55.890534+0000","last_undegraded":"2024-03-26T21:29:55.890534+0000","last_fullsized":"2024-03-26T21:29:55.890534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.343773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775168+0000","last_change":"2024-03-26T21:28:40.891718+0000","last_active":"2024-03-26T21:29:40.775168+0000","last_peered":"2024-03-26T21:29:40.775168+0000","last_clean":"2024-03-26T21:29:40.775168+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:29:40.775168+0000","last_undegraded":"2024-03-26T21:29:40.775168+0000","last_fullsized":"2024-03-26T21:29:40.775168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.486662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890351+0000","last_change":"2024-03-26T21:28:38.868625+0000","last_active":"2024-03-26T21:29:55.890351+0000","last_peered":"2024-03-26T21:29:55.890351+0000","last_clean":"2024-03-26T21:29:55.890351+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:29:55.890351+0000","last_undegraded":"2024-03-26T21:29:55.890351+0000","last_fullsized":"2024-03-26T21:29:55.890351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.070691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771704+0000","last_change":"2024-03-26T21:28:34.844448+0000","last_active":"2024-03-26T21:29:40.771704+0000","last_peered":"2024-03-26T21:29:40.771704+0000","last_clean":"2024-03-26T21:29:40.771704+0000","last_became_active":"2024-03-26T21:28:34.842160+0000","last_became_peered":"2024-03-26T21:28:34.842160+0000","last_unstale":"2024-03-26T21:29:40.771704+0000","last_undegraded":"2024-03-26T21:29:40.771704+0000","last_fullsized":"2024-03-26T21:29:40.771704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.941135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771507+0000","last_change":"2024-03-26T21:28:36.846742+0000","last_active":"2024-03-26T21:29:40.771507+0000","last_peered":"2024-03-26T21:29:40.771507+0000","last_clean":"2024-03-26T21:29:40.771507+0000","last_became_active":"2024-03-26T21:28:36.846289+0000","last_became_peered":"2024-03-26T21:28:36.846289+0000","last_unstale":"2024-03-26T21:29:40.771507+0000","last_undegraded":"2024-03-26T21:29:40.771507+0000","last_fullsized":"2024-03-26T21:29:40.771507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:59.376267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891634+0000","last_change":"2024-03-26T21:28:40.891431+0000","last_active":"2024-03-26T21:29:55.891634+0000","last_peered":"2024-03-26T21:29:55.891634+0000","last_clean":"2024-03-26T21:29:55.891634+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:29:55.891634+0000","last_undegraded":"2024-03-26T21:29:55.891634+0000","last_fullsized":"2024-03-26T21:29:55.891634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891087+0000","last_change":"2024-03-26T21:29:47.419090+0000","last_active":"2024-03-26T21:29:55.891087+0000","last_peered":"2024-03-26T21:29:55.891087+0000","last_clean":"2024-03-26T21:29:55.891087+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:29:55.891087+0000","last_undegraded":"2024-03-26T21:29:55.891087+0000","last_fullsized":"2024-03-26T21:29:55.891087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:29:47.418949+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.219762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584045+0000","last_change":"2024-03-26T21:29:39.550921+0000","last_active":"2024-03-26T21:29:41.584045+0000","last_peered":"2024-03-26T21:29:41.584045+0000","last_clean":"2024-03-26T21:29:41.584045+0000","last_became_active":"2024-03-26T21:28:34.839491+0000","last_became_peered":"2024-03-26T21:28:34.839491+0000","last_unstale":"2024-03-26T21:29:41.584045+0000","last_undegraded":"2024-03-26T21:29:41.584045+0000","last_fullsized":"2024-03-26T21:29:41.584045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.089443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.733967+0000","last_change":"2024-03-26T21:29:55.733996+0000","last_active":"2024-03-26T21:29:55.733967+0000","last_peered":"2024-03-26T21:29:55.733967+0000","last_clean":"2024-03-26T21:29:55.733967+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:29:55.733967+0000","last_undegraded":"2024-03-26T21:29:55.733967+0000","last_fullsized":"2024-03-26T21:29:55.733967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:29:55.733905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.733905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.846036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772065+0000","last_change":"2024-03-26T21:28:40.881531+0000","last_active":"2024-03-26T21:29:40.772065+0000","last_peered":"2024-03-26T21:29:40.772065+0000","last_clean":"2024-03-26T21:29:40.772065+0000","last_became_active":"2024-03-26T21:28:40.881234+0000","last_became_peered":"2024-03-26T21:28:40.881234+0000","last_unstale":"2024-03-26T21:29:40.772065+0000","last_undegraded":"2024-03-26T21:29:40.772065+0000","last_fullsized":"2024-03-26T21:29:40.772065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.460159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899751+0000","last_change":"2024-03-26T21:28:38.864606+0000","last_active":"2024-03-26T21:29:49.899751+0000","last_peered":"2024-03-26T21:29:49.899751+0000","last_clean":"2024-03-26T21:29:49.899751+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:29:49.899751+0000","last_undegraded":"2024-03-26T21:29:49.899751+0000","last_fullsized":"2024-03-26T21:29:49.899751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.571305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890177+0000","last_change":"2024-03-26T21:28:34.841082+0000","last_active":"2024-03-26T21:29:55.890177+0000","last_peered":"2024-03-26T21:29:55.890177+0000","last_clean":"2024-03-26T21:29:55.890177+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:29:55.890177+0000","last_undegraded":"2024-03-26T21:29:55.890177+0000","last_fullsized":"2024-03-26T21:29:55.890177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.847025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891139+0000","last_change":"2024-03-26T21:28:37.142871+0000","last_active":"2024-03-26T21:29:55.891139+0000","last_peered":"2024-03-26T21:29:55.891139+0000","last_clean":"2024-03-26T21:29:55.891139+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:29:55.891139+0000","last_undegraded":"2024-03-26T21:29:55.891139+0000","last_fullsized":"2024-03-26T21:29:55.891139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071192+0000","last_change":"2024-03-26T21:28:40.885442+0000","last_active":"2024-03-26T21:29:41.071192+0000","last_peered":"2024-03-26T21:29:41.071192+0000","last_clean":"2024-03-26T21:29:41.071192+0000","last_became_active":"2024-03-26T21:28:40.884963+0000","last_became_peered":"2024-03-26T21:28:40.884963+0000","last_unstale":"2024-03-26T21:29:41.071192+0000","last_undegraded":"2024-03-26T21:29:41.071192+0000","last_fullsized":"2024-03-26T21:29:41.071192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.759989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776431+0000","last_change":"2024-03-26T21:28:38.864418+0000","last_active":"2024-03-26T21:29:40.776431+0000","last_peered":"2024-03-26T21:29:40.776431+0000","last_clean":"2024-03-26T21:29:40.776431+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:29:40.776431+0000","last_undegraded":"2024-03-26T21:29:40.776431+0000","last_fullsized":"2024-03-26T21:29:40.776431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.783527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583901+0000","last_change":"2024-03-26T21:28:34.843600+0000","last_active":"2024-03-26T21:29:41.583901+0000","last_peered":"2024-03-26T21:29:41.583901+0000","last_clean":"2024-03-26T21:29:41.583901+0000","last_became_active":"2024-03-26T21:28:34.843175+0000","last_became_peered":"2024-03-26T21:28:34.843175+0000","last_unstale":"2024-03-26T21:29:41.583901+0000","last_undegraded":"2024-03-26T21:29:41.583901+0000","last_fullsized":"2024-03-26T21:29:41.583901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.404783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891979+0000","last_change":"2024-03-26T21:28:37.143400+0000","last_active":"2024-03-26T21:29:55.891979+0000","last_peered":"2024-03-26T21:29:55.891979+0000","last_clean":"2024-03-26T21:29:55.891979+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:29:55.891979+0000","last_undegraded":"2024-03-26T21:29:55.891979+0000","last_fullsized":"2024-03-26T21:29:55.891979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584291+0000","last_change":"2024-03-26T21:28:41.115111+0000","last_active":"2024-03-26T21:29:41.584291+0000","last_peered":"2024-03-26T21:29:41.584291+0000","last_clean":"2024-03-26T21:29:41.584291+0000","last_became_active":"2024-03-26T21:28:41.114831+0000","last_became_peered":"2024-03-26T21:28:41.114831+0000","last_unstale":"2024-03-26T21:29:41.584291+0000","last_undegraded":"2024-03-26T21:29:41.584291+0000","last_fullsized":"2024-03-26T21:29:41.584291+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.343661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891173+0000","last_change":"2024-03-26T21:28:38.875070+0000","last_active":"2024-03-26T21:29:55.891173+0000","last_peered":"2024-03-26T21:29:55.891173+0000","last_clean":"2024-03-26T21:29:55.891173+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:29:55.891173+0000","last_undegraded":"2024-03-26T21:29:55.891173+0000","last_fullsized":"2024-03-26T21:29:55.891173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.860043+0000","last_change":"2024-03-26T21:29:53.937502+0000","last_active":"2024-03-26T21:29:56.860043+0000","last_peered":"2024-03-26T21:29:56.860043+0000","last_clean":"2024-03-26T21:29:56.860043+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:29:56.860043+0000","last_undegraded":"2024-03-26T21:29:56.860043+0000","last_fullsized":"2024-03-26T21:29:56.860043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:53.937386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:53.937386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.079593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776367+0000","last_change":"2024-03-26T21:28:59.687904+0000","last_active":"2024-03-26T21:29:40.776367+0000","last_peered":"2024-03-26T21:29:40.776367+0000","last_clean":"2024-03-26T21:29:40.776367+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:29:40.776367+0000","last_undegraded":"2024-03-26T21:29:40.776367+0000","last_fullsized":"2024-03-26T21:29:40.776367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:47.645966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890758+0000","last_change":"2024-03-26T21:28:41.114110+0000","last_active":"2024-03-26T21:29:55.890758+0000","last_peered":"2024-03-26T21:29:55.890758+0000","last_clean":"2024-03-26T21:29:55.890758+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:29:55.890758+0000","last_undegraded":"2024-03-26T21:29:55.890758+0000","last_fullsized":"2024-03-26T21:29:55.890758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.352080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":71,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584544+0000","last_change":"2024-03-26T21:28:34.843004+0000","last_active":"2024-03-26T21:29:41.584544+0000","last_peered":"2024-03-26T21:29:41.584544+0000","last_clean":"2024-03-26T21:29:41.584544+0000","last_became_active":"2024-03-26T21:28:34.842712+0000","last_became_peered":"2024-03-26T21:28:34.842712+0000","last_unstale":"2024-03-26T21:29:41.584544+0000","last_undegraded":"2024-03-26T21:29:41.584544+0000","last_fullsized":"2024-03-26T21:29:41.584544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.585264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890307+0000","last_change":"2024-03-26T21:28:37.141762+0000","last_active":"2024-03-26T21:29:55.890307+0000","last_peered":"2024-03-26T21:29:55.890307+0000","last_clean":"2024-03-26T21:29:55.890307+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:55.890307+0000","last_undegraded":"2024-03-26T21:29:55.890307+0000","last_fullsized":"2024-03-26T21:29:55.890307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899871+0000","last_change":"2024-03-26T21:28:38.874954+0000","last_active":"2024-03-26T21:29:49.899871+0000","last_peered":"2024-03-26T21:29:49.899871+0000","last_clean":"2024-03-26T21:29:49.899871+0000","last_became_active":"2024-03-26T21:28:38.874646+0000","last_became_peered":"2024-03-26T21:28:38.874646+0000","last_unstale":"2024-03-26T21:29:49.899871+0000","last_undegraded":"2024-03-26T21:29:49.899871+0000","last_fullsized":"2024-03-26T21:29:49.899871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.073289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775762+0000","last_change":"2024-03-26T21:28:40.888897+0000","last_active":"2024-03-26T21:29:40.775762+0000","last_peered":"2024-03-26T21:29:40.775762+0000","last_clean":"2024-03-26T21:29:40.775762+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:29:40.775762+0000","last_undegraded":"2024-03-26T21:29:40.775762+0000","last_fullsized":"2024-03-26T21:29:40.775762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:08.336674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:44.612784+0000","last_change":"2024-03-26T21:29:44.612815+0000","last_active":"2024-03-26T21:29:44.612784+0000","last_peered":"2024-03-26T21:29:44.612784+0000","last_clean":"2024-03-26T21:29:44.612784+0000","last_became_active":"2024-03-26T21:28:34.848110+0000","last_became_peered":"2024-03-26T21:28:34.848110+0000","last_unstale":"2024-03-26T21:29:44.612784+0000","last_undegraded":"2024-03-26T21:29:44.612784+0000","last_fullsized":"2024-03-26T21:29:44.612784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.133060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:42.608528+0000","last_change":"2024-03-26T21:29:42.608574+0000","last_active":"2024-03-26T21:29:42.608528+0000","last_peered":"2024-03-26T21:29:42.608528+0000","last_clean":"2024-03-26T21:29:42.608528+0000","last_became_active":"2024-03-26T21:28:36.847905+0000","last_became_peered":"2024-03-26T21:28:36.847905+0000","last_unstale":"2024-03-26T21:29:42.608528+0000","last_undegraded":"2024-03-26T21:29:42.608528+0000","last_fullsized":"2024-03-26T21:29:42.608528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.811581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.070727+0000","last_change":"2024-03-26T21:28:38.867409+0000","last_active":"2024-03-26T21:29:41.070727+0000","last_peered":"2024-03-26T21:29:41.070727+0000","last_clean":"2024-03-26T21:29:41.070727+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:29:41.070727+0000","last_undegraded":"2024-03-26T21:29:41.070727+0000","last_fullsized":"2024-03-26T21:29:41.070727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:54.845468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583761+0000","last_change":"2024-03-26T21:28:40.887205+0000","last_active":"2024-03-26T21:29:41.583761+0000","last_peered":"2024-03-26T21:29:41.583761+0000","last_clean":"2024-03-26T21:29:41.583761+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:29:41.583761+0000","last_undegraded":"2024-03-26T21:29:41.583761+0000","last_fullsized":"2024-03-26T21:29:41.583761+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.254459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071013+0000","last_change":"2024-03-26T21:28:38.869650+0000","last_active":"2024-03-26T21:29:41.071013+0000","last_peered":"2024-03-26T21:29:41.071013+0000","last_clean":"2024-03-26T21:29:41.071013+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:29:41.071013+0000","last_undegraded":"2024-03-26T21:29:41.071013+0000","last_fullsized":"2024-03-26T21:29:41.071013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:56.021948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.625690+0000","last_change":"2024-03-26T21:29:48.625718+0000","last_active":"2024-03-26T21:29:48.625690+0000","last_peered":"2024-03-26T21:29:48.625690+0000","last_clean":"2024-03-26T21:29:48.625690+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:29:48.625690+0000","last_undegraded":"2024-03-26T21:29:48.625690+0000","last_fullsized":"2024-03-26T21:29:48.625690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.625595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.625595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.303805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:58.792845+0000","last_change":"2024-03-26T21:29:58.792883+0000","last_active":"2024-03-26T21:29:58.792845+0000","last_peered":"2024-03-26T21:29:58.792845+0000","last_clean":"2024-03-26T21:29:58.792845+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:29:58.792845+0000","last_undegraded":"2024-03-26T21:29:58.792845+0000","last_fullsized":"2024-03-26T21:29:58.792845+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:58.792783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.792783+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:02.267914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891027+0000","last_change":"2024-03-26T21:28:40.889290+0000","last_active":"2024-03-26T21:29:55.891027+0000","last_peered":"2024-03-26T21:29:55.891027+0000","last_clean":"2024-03-26T21:29:55.891027+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:29:55.891027+0000","last_undegraded":"2024-03-26T21:29:55.891027+0000","last_fullsized":"2024-03-26T21:29:55.891027+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.738713+0000","last_change":"2024-03-26T21:29:48.738742+0000","last_active":"2024-03-26T21:29:48.738713+0000","last_peered":"2024-03-26T21:29:48.738713+0000","last_clean":"2024-03-26T21:29:48.738713+0000","last_became_active":"2024-03-26T21:28:38.866104+0000","last_became_peered":"2024-03-26T21:28:38.866104+0000","last_unstale":"2024-03-26T21:29:48.738713+0000","last_undegraded":"2024-03-26T21:29:48.738713+0000","last_fullsized":"2024-03-26T21:29:48.738713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.530253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890845+0000","last_change":"2024-03-26T21:28:34.841686+0000","last_active":"2024-03-26T21:29:55.890845+0000","last_peered":"2024-03-26T21:29:55.890845+0000","last_clean":"2024-03-26T21:29:55.890845+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:29:55.890845+0000","last_undegraded":"2024-03-26T21:29:55.890845+0000","last_fullsized":"2024-03-26T21:29:55.890845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.267735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891511+0000","last_change":"2024-03-26T21:28:36.845693+0000","last_active":"2024-03-26T21:29:55.891511+0000","last_peered":"2024-03-26T21:29:55.891511+0000","last_clean":"2024-03-26T21:29:55.891511+0000","last_became_active":"2024-03-26T21:28:36.845358+0000","last_became_peered":"2024-03-26T21:28:36.845358+0000","last_unstale":"2024-03-26T21:29:55.891511+0000","last_undegraded":"2024-03-26T21:29:55.891511+0000","last_fullsized":"2024-03-26T21:29:55.891511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.265401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:42.726393+0000","last_change":"2024-03-26T21:29:42.726419+0000","last_active":"2024-03-26T21:29:42.726393+0000","last_peered":"2024-03-26T21:29:42.726393+0000","last_clean":"2024-03-26T21:29:42.726393+0000","last_became_active":"2024-03-26T21:28:40.884694+0000","last_became_peered":"2024-03-26T21:28:40.884694+0000","last_unstale":"2024-03-26T21:29:42.726393+0000","last_undegraded":"2024-03-26T21:29:42.726393+0000","last_fullsized":"2024-03-26T21:29:42.726393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.138620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890468+0000","last_change":"2024-03-26T21:28:38.867924+0000","last_active":"2024-03-26T21:29:55.890468+0000","last_peered":"2024-03-26T21:29:55.890468+0000","last_clean":"2024-03-26T21:29:55.890468+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:29:55.890468+0000","last_undegraded":"2024-03-26T21:29:55.890468+0000","last_fullsized":"2024-03-26T21:29:55.890468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771400+0000","last_change":"2024-03-26T21:28:34.842465+0000","last_active":"2024-03-26T21:29:40.771400+0000","last_peered":"2024-03-26T21:29:40.771400+0000","last_clean":"2024-03-26T21:29:40.771400+0000","last_became_active":"2024-03-26T21:28:34.842130+0000","last_became_peered":"2024-03-26T21:28:34.842130+0000","last_unstale":"2024-03-26T21:29:40.771400+0000","last_undegraded":"2024-03-26T21:29:40.771400+0000","last_fullsized":"2024-03-26T21:29:40.771400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.625559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890594+0000","last_change":"2024-03-26T21:28:59.692512+0000","last_active":"2024-03-26T21:29:55.890594+0000","last_peered":"2024-03-26T21:29:55.890594+0000","last_clean":"2024-03-26T21:29:55.890594+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:29:55.890594+0000","last_undegraded":"2024-03-26T21:29:55.890594+0000","last_fullsized":"2024-03-26T21:29:55.890594+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890365+0000","last_change":"2024-03-26T21:28:38.873506+0000","last_active":"2024-03-26T21:29:55.890365+0000","last_peered":"2024-03-26T21:29:55.890365+0000","last_clean":"2024-03-26T21:29:55.890365+0000","last_became_active":"2024-03-26T21:28:38.873196+0000","last_became_peered":"2024-03-26T21:28:38.873196+0000","last_unstale":"2024-03-26T21:29:55.890365+0000","last_undegraded":"2024-03-26T21:29:55.890365+0000","last_fullsized":"2024-03-26T21:29:55.890365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584559+0000","last_change":"2024-03-26T21:28:41.114073+0000","last_active":"2024-03-26T21:29:41.584559+0000","last_peered":"2024-03-26T21:29:41.584559+0000","last_clean":"2024-03-26T21:29:41.584559+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:29:41.584559+0000","last_undegraded":"2024-03-26T21:29:41.584559+0000","last_fullsized":"2024-03-26T21:29:41.584559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:46.228003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889543+0000","last_change":"2024-03-26T21:29:36.249432+0000","last_active":"2024-03-26T21:29:55.889543+0000","last_peered":"2024-03-26T21:29:55.889543+0000","last_clean":"2024-03-26T21:29:55.889543+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:29:55.889543+0000","last_undegraded":"2024-03-26T21:29:55.889543+0000","last_fullsized":"2024-03-26T21:29:55.889543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:36.249302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.249302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:39.117721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.695403+0000","last_change":"2024-03-26T21:29:41.695430+0000","last_active":"2024-03-26T21:29:41.695403+0000","last_peered":"2024-03-26T21:29:41.695403+0000","last_clean":"2024-03-26T21:29:41.695403+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:29:41.695403+0000","last_undegraded":"2024-03-26T21:29:41.695403+0000","last_fullsized":"2024-03-26T21:29:41.695403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.695322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.695322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.842596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891920+0000","last_change":"2024-03-26T21:29:55.054708+0000","last_active":"2024-03-26T21:29:55.891920+0000","last_peered":"2024-03-26T21:29:55.891920+0000","last_clean":"2024-03-26T21:29:55.891920+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:29:55.891920+0000","last_undegraded":"2024-03-26T21:29:55.891920+0000","last_fullsized":"2024-03-26T21:29:55.891920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.054601+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.383564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071241+0000","last_change":"2024-03-26T21:28:38.874086+0000","last_active":"2024-03-26T21:29:41.071241+0000","last_peered":"2024-03-26T21:29:41.071241+0000","last_clean":"2024-03-26T21:29:41.071241+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:29:41.071241+0000","last_undegraded":"2024-03-26T21:29:41.071241+0000","last_fullsized":"2024-03-26T21:29:41.071241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.551108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891287+0000","last_change":"2024-03-26T21:28:40.887390+0000","last_active":"2024-03-26T21:29:55.891287+0000","last_peered":"2024-03-26T21:29:55.891287+0000","last_clean":"2024-03-26T21:29:55.891287+0000","last_became_active":"2024-03-26T21:28:40.886965+0000","last_became_peered":"2024-03-26T21:28:40.886965+0000","last_unstale":"2024-03-26T21:29:55.891287+0000","last_undegraded":"2024-03-26T21:29:55.891287+0000","last_fullsized":"2024-03-26T21:29:55.891287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":304,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.583887+0000","last_change":"2024-03-26T21:28:42.275445+0000","last_active":"2024-03-26T21:29:41.583887+0000","last_peered":"2024-03-26T21:29:41.583887+0000","last_clean":"2024-03-26T21:29:41.583887+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:29:41.583887+0000","last_undegraded":"2024-03-26T21:29:41.583887+0000","last_fullsized":"2024-03-26T21:29:41.583887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T21:28:42.275350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:28:42.275350+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:52.670384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775577+0000","last_change":"2024-03-26T21:29:38.690870+0000","last_active":"2024-03-26T21:29:40.775577+0000","last_peered":"2024-03-26T21:29:40.775577+0000","last_clean":"2024-03-26T21:29:40.775577+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:29:40.775577+0000","last_undegraded":"2024-03-26T21:29:40.775577+0000","last_fullsized":"2024-03-26T21:29:40.775577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.690779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.690779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.230074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"88'24","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.860894+0000","last_change":"2024-03-26T21:29:55.903181+0000","last_active":"2024-03-26T21:29:56.860894+0000","last_peered":"2024-03-26T21:29:56.860894+0000","last_clean":"2024-03-26T21:29:56.860894+0000","last_became_active":"2024-03-26T21:28:37.140774+0000","last_became_peered":"2024-03-26T21:28:37.140774+0000","last_unstale":"2024-03-26T21:29:56.860894+0000","last_undegraded":"2024-03-26T21:29:56.860894+0000","last_fullsized":"2024-03-26T21:29:56.860894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:13.673385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584298+0000","last_change":"2024-03-26T21:28:38.868716+0000","last_active":"2024-03-26T21:29:41.584298+0000","last_peered":"2024-03-26T21:29:41.584298+0000","last_clean":"2024-03-26T21:29:41.584298+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:29:41.584298+0000","last_undegraded":"2024-03-26T21:29:41.584298+0000","last_fullsized":"2024-03-26T21:29:41.584298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.403849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891271+0000","last_change":"2024-03-26T21:28:40.887424+0000","last_active":"2024-03-26T21:29:55.891271+0000","last_peered":"2024-03-26T21:29:55.891271+0000","last_clean":"2024-03-26T21:29:55.891271+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:29:55.891271+0000","last_undegraded":"2024-03-26T21:29:55.891271+0000","last_fullsized":"2024-03-26T21:29:55.891271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:05.086375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891131+0000","last_change":"2024-03-26T21:28:34.843666+0000","last_active":"2024-03-26T21:29:55.891131+0000","last_peered":"2024-03-26T21:29:55.891131+0000","last_clean":"2024-03-26T21:29:55.891131+0000","last_became_active":"2024-03-26T21:28:34.843392+0000","last_became_peered":"2024-03-26T21:28:34.843392+0000","last_unstale":"2024-03-26T21:29:55.891131+0000","last_undegraded":"2024-03-26T21:29:55.891131+0000","last_fullsized":"2024-03-26T21:29:55.891131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:54.465645+0000","last_change":"2024-03-26T21:28:38.848730+0000","last_active":"2024-03-26T21:29:54.465645+0000","last_peered":"2024-03-26T21:29:54.465645+0000","last_clean":"2024-03-26T21:29:54.465645+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:29:54.465645+0000","last_undegraded":"2024-03-26T21:29:54.465645+0000","last_fullsized":"2024-03-26T21:29:54.465645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891745+0000","last_change":"2024-03-26T21:29:38.920799+0000","last_active":"2024-03-26T21:29:55.891745+0000","last_peered":"2024-03-26T21:29:55.891745+0000","last_clean":"2024-03-26T21:29:55.891745+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:29:55.891745+0000","last_undegraded":"2024-03-26T21:29:55.891745+0000","last_fullsized":"2024-03-26T21:29:55.891745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:38.920701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.920701+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.589043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891510+0000","last_change":"2024-03-26T21:28:38.870606+0000","last_active":"2024-03-26T21:29:55.891510+0000","last_peered":"2024-03-26T21:29:55.891510+0000","last_clean":"2024-03-26T21:29:55.891510+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:29:55.891510+0000","last_undegraded":"2024-03-26T21:29:55.891510+0000","last_fullsized":"2024-03-26T21:29:55.891510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:07.288276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889655+0000","last_change":"2024-03-26T21:28:41.116399+0000","last_active":"2024-03-26T21:29:55.889655+0000","last_peered":"2024-03-26T21:29:55.889655+0000","last_clean":"2024-03-26T21:29:55.889655+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:29:55.889655+0000","last_undegraded":"2024-03-26T21:29:55.889655+0000","last_fullsized":"2024-03-26T21:29:55.889655+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.016224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891518+0000","last_change":"2024-03-26T21:29:42.167843+0000","last_active":"2024-03-26T21:29:55.891518+0000","last_peered":"2024-03-26T21:29:55.891518+0000","last_clean":"2024-03-26T21:29:55.891518+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:29:55.891518+0000","last_undegraded":"2024-03-26T21:29:55.891518+0000","last_fullsized":"2024-03-26T21:29:55.891518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.167719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:45.815629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":95,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891363+0000","last_change":"2024-03-26T21:29:51.863137+0000","last_active":"2024-03-26T21:29:55.891363+0000","last_peered":"2024-03-26T21:29:55.891363+0000","last_clean":"2024-03-26T21:29:55.891363+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:55.891363+0000","last_undegraded":"2024-03-26T21:29:55.891363+0000","last_fullsized":"2024-03-26T21:29:55.891363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:51.862960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.862960+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:12.041689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.860621+0000","last_change":"2024-03-26T21:29:52.956781+0000","last_active":"2024-03-26T21:29:56.860621+0000","last_peered":"2024-03-26T21:29:56.860621+0000","last_clean":"2024-03-26T21:29:56.860621+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:29:56.860621+0000","last_undegraded":"2024-03-26T21:29:56.860621+0000","last_fullsized":"2024-03-26T21:29:56.860621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.956669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.956669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:54.531858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.889092+0000","last_change":"2024-03-26T21:28:40.890739+0000","last_active":"2024-03-26T21:29:55.889092+0000","last_peered":"2024-03-26T21:29:55.889092+0000","last_clean":"2024-03-26T21:29:55.889092+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:29:55.889092+0000","last_undegraded":"2024-03-26T21:29:55.889092+0000","last_fullsized":"2024-03-26T21:29:55.889092+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890929+0000","last_change":"2024-03-26T21:29:38.189988+0000","last_active":"2024-03-26T21:29:55.890929+0000","last_peered":"2024-03-26T21:29:55.890929+0000","last_clean":"2024-03-26T21:29:55.890929+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:29:55.890929+0000","last_undegraded":"2024-03-26T21:29:55.890929+0000","last_fullsized":"2024-03-26T21:29:55.890929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.189877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.189877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:38.248593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890438+0000","last_change":"2024-03-26T21:28:36.844629+0000","last_active":"2024-03-26T21:29:55.890438+0000","last_peered":"2024-03-26T21:29:55.890438+0000","last_clean":"2024-03-26T21:29:55.890438+0000","last_became_active":"2024-03-26T21:28:36.841759+0000","last_became_peered":"2024-03-26T21:28:36.841759+0000","last_unstale":"2024-03-26T21:29:55.890438+0000","last_undegraded":"2024-03-26T21:29:55.890438+0000","last_fullsized":"2024-03-26T21:29:55.890438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891123+0000","last_change":"2024-03-26T21:28:38.869750+0000","last_active":"2024-03-26T21:29:55.891123+0000","last_peered":"2024-03-26T21:29:55.891123+0000","last_clean":"2024-03-26T21:29:55.891123+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:29:55.891123+0000","last_undegraded":"2024-03-26T21:29:55.891123+0000","last_fullsized":"2024-03-26T21:29:55.891123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.584148+0000","last_change":"2024-03-26T21:28:41.115853+0000","last_active":"2024-03-26T21:29:41.584148+0000","last_peered":"2024-03-26T21:29:41.584148+0000","last_clean":"2024-03-26T21:29:41.584148+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:29:41.584148+0000","last_undegraded":"2024-03-26T21:29:41.584148+0000","last_fullsized":"2024-03-26T21:29:41.584148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.333646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891860+0000","last_change":"2024-03-26T21:28:34.846498+0000","last_active":"2024-03-26T21:29:55.891860+0000","last_peered":"2024-03-26T21:29:55.891860+0000","last_clean":"2024-03-26T21:29:55.891860+0000","last_became_active":"2024-03-26T21:28:34.846219+0000","last_became_peered":"2024-03-26T21:28:34.846219+0000","last_unstale":"2024-03-26T21:29:55.891860+0000","last_undegraded":"2024-03-26T21:29:55.891860+0000","last_fullsized":"2024-03-26T21:29:55.891860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771159+0000","last_change":"2024-03-26T21:28:36.840120+0000","last_active":"2024-03-26T21:29:40.771159+0000","last_peered":"2024-03-26T21:29:40.771159+0000","last_clean":"2024-03-26T21:29:40.771159+0000","last_became_active":"2024-03-26T21:28:36.839780+0000","last_became_peered":"2024-03-26T21:28:36.839780+0000","last_unstale":"2024-03-26T21:29:40.771159+0000","last_undegraded":"2024-03-26T21:29:40.771159+0000","last_fullsized":"2024-03-26T21:29:40.771159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.000016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899619+0000","last_change":"2024-03-26T21:28:38.864688+0000","last_active":"2024-03-26T21:29:49.899619+0000","last_peered":"2024-03-26T21:29:49.899619+0000","last_clean":"2024-03-26T21:29:49.899619+0000","last_became_active":"2024-03-26T21:28:38.864364+0000","last_became_peered":"2024-03-26T21:28:38.864364+0000","last_unstale":"2024-03-26T21:29:49.899619+0000","last_undegraded":"2024-03-26T21:29:49.899619+0000","last_fullsized":"2024-03-26T21:29:49.899619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:48.632429+0000","last_change":"2024-03-26T21:29:48.632454+0000","last_active":"2024-03-26T21:29:48.632429+0000","last_peered":"2024-03-26T21:29:48.632429+0000","last_clean":"2024-03-26T21:29:48.632429+0000","last_became_active":"2024-03-26T21:28:40.886497+0000","last_became_peered":"2024-03-26T21:28:40.886497+0000","last_unstale":"2024-03-26T21:29:48.632429+0000","last_undegraded":"2024-03-26T21:29:48.632429+0000","last_fullsized":"2024-03-26T21:29:48.632429+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.661591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776254+0000","last_change":"2024-03-26T21:28:34.839825+0000","last_active":"2024-03-26T21:29:40.776254+0000","last_peered":"2024-03-26T21:29:40.776254+0000","last_clean":"2024-03-26T21:29:40.776254+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:29:40.776254+0000","last_undegraded":"2024-03-26T21:29:40.776254+0000","last_fullsized":"2024-03-26T21:29:40.776254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:41.674843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775290+0000","last_change":"2024-03-26T21:28:37.143706+0000","last_active":"2024-03-26T21:29:40.775290+0000","last_peered":"2024-03-26T21:29:40.775290+0000","last_clean":"2024-03-26T21:29:40.775290+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:29:40.775290+0000","last_undegraded":"2024-03-26T21:29:40.775290+0000","last_fullsized":"2024-03-26T21:29:40.775290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.770981+0000","last_change":"2024-03-26T21:28:38.863832+0000","last_active":"2024-03-26T21:29:40.770981+0000","last_peered":"2024-03-26T21:29:40.770981+0000","last_clean":"2024-03-26T21:29:40.770981+0000","last_became_active":"2024-03-26T21:28:38.863503+0000","last_became_peered":"2024-03-26T21:28:38.863503+0000","last_unstale":"2024-03-26T21:29:40.770981+0000","last_undegraded":"2024-03-26T21:29:40.770981+0000","last_fullsized":"2024-03-26T21:29:40.770981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.090750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.890764+0000","last_change":"2024-03-26T21:28:40.891483+0000","last_active":"2024-03-26T21:29:55.890764+0000","last_peered":"2024-03-26T21:29:55.890764+0000","last_clean":"2024-03-26T21:29:55.890764+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:29:55.890764+0000","last_undegraded":"2024-03-26T21:29:55.890764+0000","last_fullsized":"2024-03-26T21:29:55.890764+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899872+0000","last_change":"2024-03-26T21:28:38.871742+0000","last_active":"2024-03-26T21:29:49.899872+0000","last_peered":"2024-03-26T21:29:49.899872+0000","last_clean":"2024-03-26T21:29:49.899872+0000","last_became_active":"2024-03-26T21:28:38.871425+0000","last_became_peered":"2024-03-26T21:28:38.871425+0000","last_unstale":"2024-03-26T21:29:49.899872+0000","last_undegraded":"2024-03-26T21:29:49.899872+0000","last_fullsized":"2024-03-26T21:29:49.899872+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:55.473329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.689412+0000","last_change":"2024-03-26T21:29:41.689438+0000","last_active":"2024-03-26T21:29:41.689412+0000","last_peered":"2024-03-26T21:29:41.689412+0000","last_clean":"2024-03-26T21:29:41.689412+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:29:41.689412+0000","last_undegraded":"2024-03-26T21:29:41.689412+0000","last_fullsized":"2024-03-26T21:29:41.689412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.689332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.689332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.552879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.892132+0000","last_change":"2024-03-26T21:28:36.848001+0000","last_active":"2024-03-26T21:29:55.892132+0000","last_peered":"2024-03-26T21:29:55.892132+0000","last_clean":"2024-03-26T21:29:55.892132+0000","last_became_active":"2024-03-26T21:28:36.847705+0000","last_became_peered":"2024-03-26T21:28:36.847705+0000","last_unstale":"2024-03-26T21:29:55.892132+0000","last_undegraded":"2024-03-26T21:29:55.892132+0000","last_fullsized":"2024-03-26T21:29:55.892132+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:03.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.775503+0000","last_change":"2024-03-26T21:28:41.113828+0000","last_active":"2024-03-26T21:29:40.775503+0000","last_peered":"2024-03-26T21:29:40.775503+0000","last_clean":"2024-03-26T21:29:40.775503+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:29:40.775503+0000","last_undegraded":"2024-03-26T21:29:40.775503+0000","last_fullsized":"2024-03-26T21:29:40.775503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:51.030485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891464+0000","last_change":"2024-03-26T21:28:38.872061+0000","last_active":"2024-03-26T21:29:55.891464+0000","last_peered":"2024-03-26T21:29:55.891464+0000","last_clean":"2024-03-26T21:29:55.891464+0000","last_became_active":"2024-03-26T21:28:38.869931+0000","last_became_peered":"2024-03-26T21:28:38.869931+0000","last_unstale":"2024-03-26T21:29:55.891464+0000","last_undegraded":"2024-03-26T21:29:55.891464+0000","last_fullsized":"2024-03-26T21:29:55.891464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.256868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:50.575688+0000","last_change":"2024-03-26T21:29:50.575715+0000","last_active":"2024-03-26T21:29:50.575688+0000","last_peered":"2024-03-26T21:29:50.575688+0000","last_clean":"2024-03-26T21:29:50.575688+0000","last_became_active":"2024-03-26T21:28:34.842398+0000","last_became_peered":"2024-03-26T21:28:34.842398+0000","last_unstale":"2024-03-26T21:29:50.575688+0000","last_undegraded":"2024-03-26T21:29:50.575688+0000","last_fullsized":"2024-03-26T21:29:50.575688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.357015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771980+0000","last_change":"2024-03-26T21:28:36.842091+0000","last_active":"2024-03-26T21:29:40.771980+0000","last_peered":"2024-03-26T21:29:40.771980+0000","last_clean":"2024-03-26T21:29:40.771980+0000","last_became_active":"2024-03-26T21:28:36.841703+0000","last_became_peered":"2024-03-26T21:28:36.841703+0000","last_unstale":"2024-03-26T21:29:40.771980+0000","last_undegraded":"2024-03-26T21:29:40.771980+0000","last_fullsized":"2024-03-26T21:29:40.771980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:51.833347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:41.071374+0000","last_change":"2024-03-26T21:28:40.885118+0000","last_active":"2024-03-26T21:29:41.071374+0000","last_peered":"2024-03-26T21:29:41.071374+0000","last_clean":"2024-03-26T21:29:41.071374+0000","last_became_active":"2024-03-26T21:28:40.884823+0000","last_became_peered":"2024-03-26T21:28:40.884823+0000","last_unstale":"2024-03-26T21:29:41.071374+0000","last_undegraded":"2024-03-26T21:29:41.071374+0000","last_fullsized":"2024-03-26T21:29:41.071374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:59.320539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T21:29:55.891719+0000","last_change":"2024-03-26T21:29:40.185232+0000","last_active":"2024-03-26T21:29:55.891719+0000","last_peered":"2024-03-26T21:29:55.891719+0000","last_clean":"2024-03-26T21:29:55.891719+0000","last_became_active":"2024-03-26T21:28:38.867110+0000","last_became_peered":"2024-03-26T21:28:38.867110+0000","last_unstale":"2024-03-26T21:29:55.891719+0000","last_undegraded":"2024-03-26T21:29:55.891719+0000","last_fullsized":"2024-03-26T21:29:55.891719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.295284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.860455+0000","last_change":"2024-03-26T21:29:39.867915+0000","last_active":"2024-03-26T21:29:56.860455+0000","last_peered":"2024-03-26T21:29:56.860455+0000","last_clean":"2024-03-26T21:29:56.860455+0000","last_became_active":"2024-03-26T21:28:34.846385+0000","last_became_peered":"2024-03-26T21:28:34.846385+0000","last_unstale":"2024-03-26T21:29:56.860455+0000","last_undegraded":"2024-03-26T21:29:56.860455+0000","last_fullsized":"2024-03-26T21:29:56.860455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:51.580336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771735+0000","last_change":"2024-03-26T21:28:36.842153+0000","last_active":"2024-03-26T21:29:40.771735+0000","last_peered":"2024-03-26T21:29:40.771735+0000","last_clean":"2024-03-26T21:29:40.771735+0000","last_became_active":"2024-03-26T21:28:36.841769+0000","last_became_peered":"2024-03-26T21:28:36.841769+0000","last_unstale":"2024-03-26T21:29:40.771735+0000","last_undegraded":"2024-03-26T21:29:40.771735+0000","last_fullsized":"2024-03-26T21:29:40.771735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.653845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.772133+0000","last_change":"2024-03-26T21:28:40.880305+0000","last_active":"2024-03-26T21:29:40.772133+0000","last_peered":"2024-03-26T21:29:40.772133+0000","last_clean":"2024-03-26T21:29:40.772133+0000","last_became_active":"2024-03-26T21:28:40.879974+0000","last_became_peered":"2024-03-26T21:28:40.879974+0000","last_unstale":"2024-03-26T21:29:40.772133+0000","last_undegraded":"2024-03-26T21:29:40.772133+0000","last_fullsized":"2024-03-26T21:29:40.772133+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:50.399077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:49.899652+0000","last_change":"2024-03-26T21:28:38.867661+0000","last_active":"2024-03-26T21:29:49.899652+0000","last_peered":"2024-03-26T21:29:49.899652+0000","last_clean":"2024-03-26T21:29:49.899652+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:29:49.899652+0000","last_undegraded":"2024-03-26T21:29:49.899652+0000","last_fullsized":"2024-03-26T21:29:49.899652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.776015+0000","last_change":"2024-03-26T21:28:34.848313+0000","last_active":"2024-03-26T21:29:40.776015+0000","last_peered":"2024-03-26T21:29:40.776015+0000","last_clean":"2024-03-26T21:29:40.776015+0000","last_became_active":"2024-03-26T21:28:34.848009+0000","last_became_peered":"2024-03-26T21:28:34.848009+0000","last_unstale":"2024-03-26T21:29:40.776015+0000","last_undegraded":"2024-03-26T21:29:40.776015+0000","last_fullsized":"2024-03-26T21:29:40.776015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.325454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T21:29:40.771552+0000","last_change":"2024-03-26T21:29:37.647253+0000","last_active":"2024-03-26T21:29:40.771552+0000","last_peered":"2024-03-26T21:29:40.771552+0000","last_clean":"2024-03-26T21:29:40.771552+0000","last_became_active":"2024-03-26T21:28:36.847902+0000","last_became_peered":"2024-03-26T21:28:36.847902+0000","last_unstale":"2024-03-26T21:29:40.771552+0000","last_undegraded":"2024-03-26T21:29:40.771552+0000","last_fullsized":"2024-03-26T21:29:40.771552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.419033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716776,"statfs":{"total":95995035648,"available":95965978624,"internally_reserved":0,"allocated":1392640,"data_stored":937448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364828,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":948054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":831488,"data_stored":353090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1392640,"data_stored":936754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":761856,"data_stored":344783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":345572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":346620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":786432,"data_stored":345352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:30:04.054 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T21:30:04.054 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T21:30:04.054 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T21:30:04.054 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T21:30:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:30:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:30:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:30:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:30:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:30:04.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:30:04.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:30:04.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:03 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:30:04.303 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:30:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:30:04.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:30:04.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:30:04.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:03 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:30:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:30:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:04 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:30:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:30:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:30:05.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:04 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.f deep-scrub starts 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.f deep-scrub ok 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:30:06.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.f deep-scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.f deep-scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:30:06.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:30:06.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:30:06.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:05 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:30:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:30:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:30:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:30:06.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.f deep-scrub starts 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.f deep-scrub ok 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:30:06.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:05 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:30:06.243 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:30:06.243 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T21:30:06.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:06.878 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T21:30:06.879 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T21:30:06.879 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-26T21:30:06.890 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T21:30:06.891 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-testing-2024-03-26-0925'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-testing-2024-03-26-0925', 'sha1': '42fc52b86b14d4f88c40d75eb7102e0ab3918f55'}} 2024-03-26T21:30:06.891 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T21:30:06.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-26T21:30:07.135 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:30:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:30:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:30:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:30:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:30:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2689813961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:30:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:30:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:06 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:30:07.256 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:07.256 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:30:07.256 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:30:07.256 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2689813961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2689813961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T21:30:07.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:30:07.258 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:06 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:30:07.815 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:30:07.737Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002109887s 2024-03-26T21:30:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:30:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:30:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:30:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:30:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:30:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:07 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:30:08.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:30:08.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:07 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:30:08.903 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 systemd[1]: Started Ceph grafana.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:09.117 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.118 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:30:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:09.119 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE Bus STOPPING 2024-03-26T21:30:09.153 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:09.153 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:30:09.153 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:30:09.153 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:30:09.153 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:30:09.154 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.154 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.154 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.154 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:09.154 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910672122Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T21:30:09.154 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910847564Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T21:30:09.154 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910908893Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910918726Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910924087Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910929001Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910933621Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910938635Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910943951Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910949173Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910954337Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910959952Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910964831Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.91096993Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910974861Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T21:30:09.155 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910979325Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910983816Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910990723Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=settings t=2024-03-26T21:30:08.910995592Z level=info msg="App mode production" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore t=2024-03-26T21:30:08.911063594Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore t=2024-03-26T21:30:08.911079772Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.911902215Z level=info msg="Starting DB migrations" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.911978825Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.912515844Z level=info msg="Executing migration" id="create user table" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.912924688Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.913388485Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.913806664Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.914412493Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T21:30:09.156 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.91641268Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.919126113Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.919536819Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.922437794Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.92309572Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.925116171Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.925512133Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.925934875Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.926252533Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T21:30:09.157 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.926673983Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.926997271Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.927444684Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.927831824Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.928291113Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.929728505Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.930135058Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.930563309Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.930969932Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.931397128Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.933972125Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T21:30:09.159 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.936396624Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.939184859Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.941603994Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.944211025Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.946688146Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.949068294Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.950376425Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.953241741Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.955819316Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.95848028Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.961235285Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.963724242Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.966473644Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.966931594Z level=info msg="Executing migration" id="create star table" 2024-03-26T21:30:09.160 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.967771262Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.968579731Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.969390572Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.970208674Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.97121847Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.971899342Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.97231555Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.973161134Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.973998136Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.974862791Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.975192949Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.976056926Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.977053725Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.977766971Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.978188398Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.978984801Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T21:30:09.161 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.979960034Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.981236097Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.981637084Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.982500146Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.983448394Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.98418973Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.984671089Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.985532614Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.986078878Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.98659262Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.987407653Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.987840646Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.98836396Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.98932508Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T21:30:09.162 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.990092094Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.990928Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.991252792Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.992164151Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.995082008Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:08.997699034Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.000711999Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.003333889Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.00610042Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.008980173Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.01160639Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.01427208Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.017118064Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.020069561Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T21:30:09.163 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.022847996Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.025579468Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.028578048Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.03125165Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.034001153Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.036831646Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.039497376Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.042187696Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.044842282Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.047658603Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.050241569Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.053020881Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.0554645Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.058314184Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.060684871Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T21:30:09.164 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.063213919Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.065791894Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.068479812Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.071291665Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.073848562Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.076625414Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.079334405Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.081758052Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.084416867Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.087148921Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.087911702Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.088801393Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.089755031Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.090471682Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.091418449Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T21:30:09.165 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.092155525Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.093018682Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.093956332Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.094688581Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.095563078Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.096480213Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.097297405Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.098275161Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.098992159Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.100270584Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.101149415Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.102125669Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.102844953Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.103254886Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T21:30:09.166 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.104237357Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.10498744Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.105958398Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.106650304Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.107659398Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.108331868Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.109334506Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.109970184Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.110954148Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.111693596Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.112481221Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.112894326Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.113328305Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.114173628Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T21:30:09.167 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.115051551Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.11545793Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.118198302Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.121160879Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.123877373Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.126500577Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.129383125Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.132132853Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.135110766Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.135670583Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.136308367Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.136600424Z level=info msg="Executing migration" id="create session table" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.137038784Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.137345969Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T21:30:09.168 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.137618487Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.138011001Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.1384185Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.138679339Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.138943303Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.139245476Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.139543578Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.139933517Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.140228624Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.140869754Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.14128279Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.141909157Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.142579752Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.142879028Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T21:30:09.169 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.143325378Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.14379644Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.14457755Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.145476424Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.146119716Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.147106829Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.147672455Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.148456062Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.149267151Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.150988988Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.151767776Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.152560921Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.153333863Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T21:30:09.170 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.154166317Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.154949085Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.155715784Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.156505582Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.157324675Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.158124504Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.158942396Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.159838389Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.160551385Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.161410365Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.162165909Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.162994253Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.163828677Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.164612725Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T21:30:09.171 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.165688302Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.166760044Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.167617417Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.168428449Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.169191472Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.170262797Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T21:30:09.406 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.172875533Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.175602267Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.178487401Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.181344816Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.183915652Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.186590574Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.189359991Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.192090015Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.194796361Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.197480324Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.200395163Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.203038523Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.20592338Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T21:30:09.407 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.208357321Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.211132052Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.213694808Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.21657768Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.219289256Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.222030847Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.224838868Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.22729576Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.229851737Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.232506129Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.234888577Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.237472662Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.239878899Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T21:30:09.409 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.242477086Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.245058333Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.247737429Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.250449622Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.252890458Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.255545639Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.258145613Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.260356901Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.263234035Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.26390708Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.264687989Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.265411541Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.266189812Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T21:30:09.410 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.267041794Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.267561065Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.268353382Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.269125431Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.269997631Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.270681852Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.271518599Z level=info msg="Executing migration" id="create team table" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.272132822Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.27257783Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.273372285Z level=info msg="Executing migration" id="create team member table" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.274168455Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.274972358Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.275882007Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.276578686Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.277427113Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T21:30:09.411 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.278262811Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.279106162Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.279887011Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.280800383Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.28141291Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.281852511Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.282621004Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.283361969Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.283790924Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.284212207Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.284624965Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.284891921Z level=info msg="Executing migration" id="create tag table" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.28527948Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.285702548Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T21:30:09.412 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.286089221Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.286516766Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.286919178Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.289285997Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.289685265Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.290118207Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.290429532Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.290757658Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.291104964Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.291549998Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.29180729Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.292628533Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.293494525Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.294387376Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.295272855Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T21:30:09.413 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.295711577Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.296591162Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.299554821Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.302382145Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.305154913Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.305708231Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.308659132Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.311394197Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.314134959Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.316939817Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.319646505Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.322577873Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.325361571Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.325776715Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T21:30:09.414 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.328779686Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.329332809Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.331581639Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.333879862Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.334205094Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.336995822Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.339469834Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.342229744Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.342685341Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.345536874Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.348312105Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.350627816Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.351071777Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.353886757Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T21:30:09.415 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.356510244Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.35922994Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.359945397Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.362468775Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.365095609Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.365835814Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.366739967Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.367514907Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.368327897Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.372873535Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.377361156Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.378231571Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.37905875Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.380066541Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T21:30:09.416 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.380536259Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.381364522Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.382210814Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.383080091Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.383382091Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.384355098Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.385315674Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.386248068Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.38668947Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.387541275Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.388486033Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.38943805Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.390263622Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.391213776Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.392156974Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T21:30:09.417 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.393154767Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.393879896Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.39436463Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.395304996Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.396069866Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.397069793Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.39802266Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.398987683Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.39995322Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T21:30:09.418 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.400918549Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.401237512Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.404263619Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.40688825Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.409741314Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.410733739Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.413520172Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T21:30:09.419 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.416222862Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T21:30:09.449 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:30:09.450 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE Bus STOPPED 2024-03-26T21:30:09.450 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE Bus STARTING 2024-03-26T21:30:09.450 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE Serving on http://:::9283 2024-03-26T21:30:09.450 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:09] ENGINE Bus STARTED 2024-03-26T21:30:09.739 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.419010307Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.419607426Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.422476624Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.425340579Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.42792596Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.428336296Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.431166449Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.434132588Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.436828112Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.437275006Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T21:30:09.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.437350579Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.440192457Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.443009934Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.445536708Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.445674785Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.445789326Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.446205752Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.448941154Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.451733255Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.453997072Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.454484883Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.457275986Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.459735337Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.462303628Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T21:30:09.741 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.462770017Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.465349722Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.471020423Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.473900437Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.474275442Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.479990571Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.485712474Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.488392996Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.488831709Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.489707788Z level=info msg="Executing migration" id="create permission table" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.490696461Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.491376272Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T21:30:09.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.49180988Z level=info msg="Executing migration" id="create role table" 2024-03-26T21:30:09.746 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.49277986Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T21:30:09.746 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.494012635Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T21:30:09.746 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.495052113Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T21:30:09.746 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.495502809Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T21:30:09.746 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.496332338Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.497182007Z level=info msg="Executing migration" id="create team role table" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.49811734Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.498565313Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.499418796Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.500250166Z level=info msg="Executing migration" id="create user role table" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.501218848Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.501670636Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.502552973Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.503405005Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.504250103Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.504690238Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.505538012Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.506648522Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T21:30:09.747 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.50749699Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.507927569Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.508779343Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.509627402Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.51047146Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.510922652Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.512267851Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.512658085Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.513086413Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.513375008Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.513755539Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.514221515Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.514609761Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T21:30:09.748 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.514937882Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.515360254Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.515789673Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.516259714Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.51744796Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.517790666Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.518206383Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.52086691Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.521360046Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.522489724Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.522892558Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.52329969Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.523379244Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.52364564Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:30:09.749 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.523721809Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.523958215Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.524389002Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.524802897Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.527539718Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.530149038Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.533006758Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.533492759Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.536360057Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.536874239Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.539015116Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.541760525Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.542247022Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.546089231Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T21:30:09.750 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.548711308Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.551138828Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.551459542Z level=info msg="Executing migration" id="add share column" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.55411862Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.556543941Z level=info msg="Executing migration" id="create file table" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.559037448Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.559494358Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.562063241Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.562453765Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.565083729Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.567648147Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.570286358Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.572801587Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.575484941Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T21:30:09.751 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.576864819Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.579596766Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.580404376Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.58110229Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.581436328Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.582199475Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.5834043Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.584585874Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.585368223Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.598427773Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.599156782Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.599908653Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.604247943Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.604993247Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.605768729Z level=info msg="Executing migration" id="create folder table" 2024-03-26T21:30:09.752 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.606540771Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.607316017Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.607872775Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.608769652Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=migrator t=2024-03-26T21:30:09.609415375Z level=info msg="migrations completed" performed=485 skipped=0 duration=697.45661ms 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore t=2024-03-26T21:30:09.609791103Z level=info msg="Created default organization" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=plugin.finder t=2024-03-26T21:30:09.643550519Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=secrets t=2024-03-26T21:30:09.643842969Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=query_data t=2024-03-26T21:30:09.646000388Z level=info msg="Query Service initialization" 2024-03-26T21:30:09.753 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=live.push_http t=2024-03-26T21:30:09.648415033Z level=info msg="Live Push Gateway initialization" 2024-03-26T21:30:09.785 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-26T21:30:09.785 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T21:30:09.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:30:10.046 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.9 deep-scrub starts 2024-03-26T21:30:10.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 6.9 deep-scrub ok 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: Deploying daemon node-exporter.a on smithi055 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.9 deep-scrub starts 2024-03-26T21:30:10.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 6.9 deep-scrub ok 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: Deploying daemon node-exporter.a on smithi055 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:30:10.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:09 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:30:10.279 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.9 deep-scrub starts 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 6.9 deep-scrub ok 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: Deploying daemon node-exporter.a on smithi055 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:30:10.280 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:09 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:30:10.739 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=infra.usagestats.collector t=2024-03-26T21:30:10.287642271Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T21:30:10.739 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=provisioning.datasources t=2024-03-26T21:30:10.288798173Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T21:30:10.739 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=provisioning.datasources t=2024-03-26T21:30:10.300068158Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=provisioning.alerting t=2024-03-26T21:30:10.310812093Z level=info msg="starting to provision alerting" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=provisioning.alerting t=2024-03-26T21:30:10.310822946Z level=info msg="finished to provision alerting" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=grafanaStorageLogger t=2024-03-26T21:30:10.310894646Z level=info msg="storage starting" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=ngalert.state.manager t=2024-03-26T21:30:10.310919741Z level=info msg="Warming state cache for startup" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=ngalert.state.manager t=2024-03-26T21:30:10.311299139Z level=info msg="State cache has been initialized" states=0 duration=379.176µs 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=ticker t=2024-03-26T21:30:10.311489032Z level=info msg=starting first_tick=2024-03-26T21:30:20Z 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=ngalert.multiorg.alertmanager t=2024-03-26T21:30:10.311574161Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=http.server t=2024-03-26T21:30:10.319413318Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore.transactions t=2024-03-26T21:30:10.331541387Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore.transactions t=2024-03-26T21:30:10.341754661Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T21:30:10.740 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:30:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=sqlstore.transactions t=2024-03-26T21:30:10.352079473Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-26T21:30:11.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:11 smithi067 ceph-mon[33346]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:11.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:11 smithi067 ceph-mon[33346]: 4.12 deep-scrub starts 2024-03-26T21:30:11.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:11 smithi067 ceph-mon[33346]: 4.12 deep-scrub ok 2024-03-26T21:30:11.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:11 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[29406]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[29406]: 4.12 deep-scrub starts 2024-03-26T21:30:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[29406]: 4.12 deep-scrub ok 2024-03-26T21:30:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[40587]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[40587]: 4.12 deep-scrub starts 2024-03-26T21:30:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[40587]: 4.12 deep-scrub ok 2024-03-26T21:30:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:11 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:12.168 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:30:12.171 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:30:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:30:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:12 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:30:12.317 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:12 smithi055 bash[76226]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T21:30:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:30:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:30:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:12 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:30:13.215 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-26T21:30:11.282719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1019,"num_read_kb":865,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240800,"kb_used_data":7908,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720416,"statfs":{"total":767960285184,"available":767713705984,"internally_reserved":0,"allocated":8097792,"data_stored":4569006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002766"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:03.491319+0000","last_change":"2024-03-26T21:30:03.491344+0000","last_active":"2024-03-26T21:30:03.491319+0000","last_peered":"2024-03-26T21:30:03.491319+0000","last_clean":"2024-03-26T21:30:03.491319+0000","last_became_active":"2024-03-26T21:28:40.879971+0000","last_became_peered":"2024-03-26T21:28:40.879971+0000","last_unstale":"2024-03-26T21:30:03.491319+0000","last_undegraded":"2024-03-26T21:30:03.491319+0000","last_fullsized":"2024-03-26T21:30:03.491319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.372263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985743+0000","last_change":"2024-03-26T21:28:38.866679+0000","last_active":"2024-03-26T21:30:05.985743+0000","last_peered":"2024-03-26T21:30:05.985743+0000","last_clean":"2024-03-26T21:30:05.985743+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:30:05.985743+0000","last_undegraded":"2024-03-26T21:30:05.985743+0000","last_fullsized":"2024-03-26T21:30:05.985743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500974+0000","last_change":"2024-03-26T21:28:34.844384+0000","last_active":"2024-03-26T21:29:56.500974+0000","last_peered":"2024-03-26T21:29:56.500974+0000","last_clean":"2024-03-26T21:29:56.500974+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:29:56.500974+0000","last_undegraded":"2024-03-26T21:29:56.500974+0000","last_fullsized":"2024-03-26T21:29:56.500974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500209+0000","last_change":"2024-03-26T21:29:51.544731+0000","last_active":"2024-03-26T21:29:56.500209+0000","last_peered":"2024-03-26T21:29:56.500209+0000","last_clean":"2024-03-26T21:29:56.500209+0000","last_became_active":"2024-03-26T21:28:36.841677+0000","last_became_peered":"2024-03-26T21:28:36.841677+0000","last_unstale":"2024-03-26T21:29:56.500209+0000","last_undegraded":"2024-03-26T21:29:56.500209+0000","last_fullsized":"2024-03-26T21:29:56.500209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.477780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689769+0000","last_change":"2024-03-26T21:29:49.749475+0000","last_active":"2024-03-26T21:29:56.689769+0000","last_peered":"2024-03-26T21:29:56.689769+0000","last_clean":"2024-03-26T21:29:56.689769+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:29:56.689769+0000","last_undegraded":"2024-03-26T21:29:56.689769+0000","last_fullsized":"2024-03-26T21:29:56.689769+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:49.749386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:49.749386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.347098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500215+0000","last_change":"2024-03-26T21:29:45.645672+0000","last_active":"2024-03-26T21:29:56.500215+0000","last_peered":"2024-03-26T21:29:56.500215+0000","last_clean":"2024-03-26T21:29:56.500215+0000","last_became_active":"2024-03-26T21:28:38.866501+0000","last_became_peered":"2024-03-26T21:28:38.866501+0000","last_unstale":"2024-03-26T21:29:56.500215+0000","last_undegraded":"2024-03-26T21:29:56.500215+0000","last_fullsized":"2024-03-26T21:29:56.500215+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.799670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690396+0000","last_change":"2024-03-26T21:29:50.735279+0000","last_active":"2024-03-26T21:29:56.690396+0000","last_peered":"2024-03-26T21:29:56.690396+0000","last_clean":"2024-03-26T21:29:56.690396+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:29:56.690396+0000","last_undegraded":"2024-03-26T21:29:56.690396+0000","last_fullsized":"2024-03-26T21:29:56.690396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.735199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.735199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.656417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.728200+0000","last_change":"2024-03-26T21:30:05.728263+0000","last_active":"2024-03-26T21:30:05.728200+0000","last_peered":"2024-03-26T21:30:05.728200+0000","last_clean":"2024-03-26T21:30:05.728200+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:30:05.728200+0000","last_undegraded":"2024-03-26T21:30:05.728200+0000","last_fullsized":"2024-03-26T21:30:05.728200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:05.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.728097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.255731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:58.804696+0000","last_change":"2024-03-26T21:29:58.804717+0000","last_active":"2024-03-26T21:29:58.804696+0000","last_peered":"2024-03-26T21:29:58.804696+0000","last_clean":"2024-03-26T21:29:58.804696+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:29:58.804696+0000","last_undegraded":"2024-03-26T21:29:58.804696+0000","last_fullsized":"2024-03-26T21:29:58.804696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.804636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.804636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.335495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142031+0000","last_change":"2024-03-26T21:28:38.872667+0000","last_active":"2024-03-26T21:29:57.142031+0000","last_peered":"2024-03-26T21:29:57.142031+0000","last_clean":"2024-03-26T21:29:57.142031+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:29:57.142031+0000","last_undegraded":"2024-03-26T21:29:57.142031+0000","last_fullsized":"2024-03-26T21:29:57.142031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.510486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142058+0000","last_change":"2024-03-26T21:28:34.842302+0000","last_active":"2024-03-26T21:29:57.142058+0000","last_peered":"2024-03-26T21:29:57.142058+0000","last_clean":"2024-03-26T21:29:57.142058+0000","last_became_active":"2024-03-26T21:28:34.841544+0000","last_became_peered":"2024-03-26T21:28:34.841544+0000","last_unstale":"2024-03-26T21:29:57.142058+0000","last_undegraded":"2024-03-26T21:29:57.142058+0000","last_fullsized":"2024-03-26T21:29:57.142058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.708588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500522+0000","last_change":"2024-03-26T21:29:52.496860+0000","last_active":"2024-03-26T21:29:56.500522+0000","last_peered":"2024-03-26T21:29:56.500522+0000","last_clean":"2024-03-26T21:29:56.500522+0000","last_became_active":"2024-03-26T21:28:36.847804+0000","last_became_peered":"2024-03-26T21:28:36.847804+0000","last_unstale":"2024-03-26T21:29:56.500522+0000","last_undegraded":"2024-03-26T21:29:56.500522+0000","last_fullsized":"2024-03-26T21:29:56.500522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.032728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.141595+0000","last_change":"2024-03-26T21:28:40.887065+0000","last_active":"2024-03-26T21:29:57.141595+0000","last_peered":"2024-03-26T21:29:57.141595+0000","last_clean":"2024-03-26T21:29:57.141595+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:29:57.141595+0000","last_undegraded":"2024-03-26T21:29:57.141595+0000","last_fullsized":"2024-03-26T21:29:57.141595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980957+0000","last_change":"2024-03-26T21:28:38.864721+0000","last_active":"2024-03-26T21:30:05.980957+0000","last_peered":"2024-03-26T21:30:05.980957+0000","last_clean":"2024-03-26T21:30:05.980957+0000","last_became_active":"2024-03-26T21:28:38.864406+0000","last_became_peered":"2024-03-26T21:28:38.864406+0000","last_unstale":"2024-03-26T21:30:05.980957+0000","last_undegraded":"2024-03-26T21:30:05.980957+0000","last_fullsized":"2024-03-26T21:30:05.980957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142118+0000","last_change":"2024-03-26T21:28:34.847255+0000","last_active":"2024-03-26T21:29:57.142118+0000","last_peered":"2024-03-26T21:29:57.142118+0000","last_clean":"2024-03-26T21:29:57.142118+0000","last_became_active":"2024-03-26T21:28:34.846956+0000","last_became_peered":"2024-03-26T21:28:34.846956+0000","last_unstale":"2024-03-26T21:29:57.142118+0000","last_undegraded":"2024-03-26T21:29:57.142118+0000","last_fullsized":"2024-03-26T21:29:57.142118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.186750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501212+0000","last_change":"2024-03-26T21:28:36.846687+0000","last_active":"2024-03-26T21:29:56.501212+0000","last_peered":"2024-03-26T21:29:56.501212+0000","last_clean":"2024-03-26T21:29:56.501212+0000","last_became_active":"2024-03-26T21:28:36.846364+0000","last_became_peered":"2024-03-26T21:28:36.846364+0000","last_unstale":"2024-03-26T21:29:56.501212+0000","last_undegraded":"2024-03-26T21:29:56.501212+0000","last_fullsized":"2024-03-26T21:29:56.501212+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.374326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501270+0000","last_change":"2024-03-26T21:28:41.114040+0000","last_active":"2024-03-26T21:29:56.501270+0000","last_peered":"2024-03-26T21:29:56.501270+0000","last_clean":"2024-03-26T21:29:56.501270+0000","last_became_active":"2024-03-26T21:28:41.113734+0000","last_became_peered":"2024-03-26T21:28:41.113734+0000","last_unstale":"2024-03-26T21:29:56.501270+0000","last_undegraded":"2024-03-26T21:29:56.501270+0000","last_fullsized":"2024-03-26T21:29:56.501270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.261763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986399+0000","last_change":"2024-03-26T21:29:58.051642+0000","last_active":"2024-03-26T21:30:05.986399+0000","last_peered":"2024-03-26T21:30:05.986399+0000","last_clean":"2024-03-26T21:30:05.986399+0000","last_became_active":"2024-03-26T21:28:38.870984+0000","last_became_peered":"2024-03-26T21:28:38.870984+0000","last_unstale":"2024-03-26T21:30:05.986399+0000","last_undegraded":"2024-03-26T21:30:05.986399+0000","last_fullsized":"2024-03-26T21:30:05.986399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.051556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.051556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.391937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.708844+0000","last_change":"2024-03-26T21:30:05.708870+0000","last_active":"2024-03-26T21:30:05.708844+0000","last_peered":"2024-03-26T21:30:05.708844+0000","last_clean":"2024-03-26T21:30:05.708844+0000","last_became_active":"2024-03-26T21:28:34.846913+0000","last_became_peered":"2024-03-26T21:28:34.846913+0000","last_unstale":"2024-03-26T21:30:05.708844+0000","last_undegraded":"2024-03-26T21:30:05.708844+0000","last_fullsized":"2024-03-26T21:30:05.708844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.708767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.708767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.984555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980623+0000","last_change":"2024-03-26T21:29:45.654379+0000","last_active":"2024-03-26T21:30:05.980623+0000","last_peered":"2024-03-26T21:30:05.980623+0000","last_clean":"2024-03-26T21:30:05.980623+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:30:05.980623+0000","last_undegraded":"2024-03-26T21:30:05.980623+0000","last_fullsized":"2024-03-26T21:30:05.980623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:29:45.654238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.654238+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.970402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690428+0000","last_change":"2024-03-26T21:28:41.113863+0000","last_active":"2024-03-26T21:29:56.690428+0000","last_peered":"2024-03-26T21:29:56.690428+0000","last_clean":"2024-03-26T21:29:56.690428+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:29:56.690428+0000","last_undegraded":"2024-03-26T21:29:56.690428+0000","last_fullsized":"2024-03-26T21:29:56.690428+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.033803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500712+0000","last_change":"2024-03-26T21:29:44.616004+0000","last_active":"2024-03-26T21:29:56.500712+0000","last_peered":"2024-03-26T21:29:56.500712+0000","last_clean":"2024-03-26T21:29:56.500712+0000","last_became_active":"2024-03-26T21:28:38.867256+0000","last_became_peered":"2024-03-26T21:28:38.867256+0000","last_unstale":"2024-03-26T21:29:56.500712+0000","last_undegraded":"2024-03-26T21:29:56.500712+0000","last_fullsized":"2024-03-26T21:29:56.500712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.353334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.464900+0000","last_change":"2024-03-26T21:30:05.464925+0000","last_active":"2024-03-26T21:30:05.464900+0000","last_peered":"2024-03-26T21:30:05.464900+0000","last_clean":"2024-03-26T21:30:05.464900+0000","last_became_active":"2024-03-26T21:28:34.846583+0000","last_became_peered":"2024-03-26T21:28:34.846583+0000","last_unstale":"2024-03-26T21:30:05.464900+0000","last_undegraded":"2024-03-26T21:30:05.464900+0000","last_fullsized":"2024-03-26T21:30:05.464900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.880478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689968+0000","last_change":"2024-03-26T21:28:36.850322+0000","last_active":"2024-03-26T21:29:56.689968+0000","last_peered":"2024-03-26T21:29:56.689968+0000","last_clean":"2024-03-26T21:29:56.689968+0000","last_became_active":"2024-03-26T21:28:36.849968+0000","last_became_peered":"2024-03-26T21:28:36.849968+0000","last_unstale":"2024-03-26T21:29:56.689968+0000","last_undegraded":"2024-03-26T21:29:56.689968+0000","last_fullsized":"2024-03-26T21:29:56.689968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:04.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981295+0000","last_change":"2024-03-26T21:29:54.614182+0000","last_active":"2024-03-26T21:30:05.981295+0000","last_peered":"2024-03-26T21:30:05.981295+0000","last_clean":"2024-03-26T21:30:05.981295+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:30:05.981295+0000","last_undegraded":"2024-03-26T21:30:05.981295+0000","last_fullsized":"2024-03-26T21:30:05.981295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:29:54.614064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:54.614064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:06.934031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:10.722267+0000","last_change":"2024-03-26T21:30:10.722294+0000","last_active":"2024-03-26T21:30:10.722267+0000","last_peered":"2024-03-26T21:30:10.722267+0000","last_clean":"2024-03-26T21:30:10.722267+0000","last_became_active":"2024-03-26T21:28:38.867245+0000","last_became_peered":"2024-03-26T21:28:38.867245+0000","last_unstale":"2024-03-26T21:30:10.722267+0000","last_undegraded":"2024-03-26T21:30:10.722267+0000","last_fullsized":"2024-03-26T21:30:10.722267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:10.722203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.722203+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:29.720986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985805+0000","last_change":"2024-03-26T21:30:01.058809+0000","last_active":"2024-03-26T21:30:05.985805+0000","last_peered":"2024-03-26T21:30:05.985805+0000","last_clean":"2024-03-26T21:30:05.985805+0000","last_became_active":"2024-03-26T21:28:34.846408+0000","last_became_peered":"2024-03-26T21:28:34.846408+0000","last_unstale":"2024-03-26T21:30:05.985805+0000","last_undegraded":"2024-03-26T21:30:05.985805+0000","last_fullsized":"2024-03-26T21:30:05.985805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.058714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.303501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501396+0000","last_change":"2024-03-26T21:28:37.143422+0000","last_active":"2024-03-26T21:29:56.501396+0000","last_peered":"2024-03-26T21:29:56.501396+0000","last_clean":"2024-03-26T21:29:56.501396+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:29:56.501396+0000","last_undegraded":"2024-03-26T21:29:56.501396+0000","last_fullsized":"2024-03-26T21:29:56.501396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:00.464940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986140+0000","last_change":"2024-03-26T21:29:45.092723+0000","last_active":"2024-03-26T21:30:05.986140+0000","last_peered":"2024-03-26T21:30:05.986140+0000","last_clean":"2024-03-26T21:30:05.986140+0000","last_became_active":"2024-03-26T21:28:40.881781+0000","last_became_peered":"2024-03-26T21:28:40.881781+0000","last_unstale":"2024-03-26T21:30:05.986140+0000","last_undegraded":"2024-03-26T21:30:05.986140+0000","last_fullsized":"2024-03-26T21:30:05.986140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.160967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.787345+0000","last_change":"2024-03-26T21:29:56.787368+0000","last_active":"2024-03-26T21:29:56.787345+0000","last_peered":"2024-03-26T21:29:56.787345+0000","last_clean":"2024-03-26T21:29:56.787345+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:29:56.787345+0000","last_undegraded":"2024-03-26T21:29:56.787345+0000","last_fullsized":"2024-03-26T21:29:56.787345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.787260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.787260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.075256+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:01.773977+0000","last_change":"2024-03-26T21:30:01.773999+0000","last_active":"2024-03-26T21:30:01.773977+0000","last_peered":"2024-03-26T21:30:01.773977+0000","last_clean":"2024-03-26T21:30:01.773977+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:30:01.773977+0000","last_undegraded":"2024-03-26T21:30:01.773977+0000","last_fullsized":"2024-03-26T21:30:01.773977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.773917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.773917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.422442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497517+0000","last_change":"2024-03-26T21:29:36.821598+0000","last_active":"2024-03-26T21:30:06.497517+0000","last_peered":"2024-03-26T21:30:06.497517+0000","last_clean":"2024-03-26T21:30:06.497517+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:30:06.497517+0000","last_undegraded":"2024-03-26T21:30:06.497517+0000","last_fullsized":"2024-03-26T21:30:06.497517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:29:36.821436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.821436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.432234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980918+0000","last_change":"2024-03-26T21:28:40.885889+0000","last_active":"2024-03-26T21:30:05.980918+0000","last_peered":"2024-03-26T21:30:05.980918+0000","last_clean":"2024-03-26T21:30:05.980918+0000","last_became_active":"2024-03-26T21:28:40.885572+0000","last_became_peered":"2024-03-26T21:28:40.885572+0000","last_unstale":"2024-03-26T21:30:05.980918+0000","last_undegraded":"2024-03-26T21:30:05.980918+0000","last_fullsized":"2024-03-26T21:30:05.980918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.749132+0000","last_change":"2024-03-26T21:30:06.749160+0000","last_active":"2024-03-26T21:30:06.749132+0000","last_peered":"2024-03-26T21:30:06.749132+0000","last_clean":"2024-03-26T21:30:06.749132+0000","last_became_active":"2024-03-26T21:28:38.870142+0000","last_became_peered":"2024-03-26T21:28:38.870142+0000","last_unstale":"2024-03-26T21:30:06.749132+0000","last_undegraded":"2024-03-26T21:30:06.749132+0000","last_fullsized":"2024-03-26T21:30:06.749132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.749046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.749046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:32.228312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500880+0000","last_change":"2024-03-26T21:29:52.893256+0000","last_active":"2024-03-26T21:29:56.500880+0000","last_peered":"2024-03-26T21:29:56.500880+0000","last_clean":"2024-03-26T21:29:56.500880+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:29:56.500880+0000","last_undegraded":"2024-03-26T21:29:56.500880+0000","last_fullsized":"2024-03-26T21:29:56.500880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.893078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.893078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.394353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.736490+0000","last_change":"2024-03-26T21:29:57.736516+0000","last_active":"2024-03-26T21:29:57.736490+0000","last_peered":"2024-03-26T21:29:57.736490+0000","last_clean":"2024-03-26T21:29:57.736490+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:29:57.736490+0000","last_undegraded":"2024-03-26T21:29:57.736490+0000","last_fullsized":"2024-03-26T21:29:57.736490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:29:57.736423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:57.736423+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.030206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689622+0000","last_change":"2024-03-26T21:28:41.117667+0000","last_active":"2024-03-26T21:29:56.689622+0000","last_peered":"2024-03-26T21:29:56.689622+0000","last_clean":"2024-03-26T21:29:56.689622+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:29:56.689622+0000","last_undegraded":"2024-03-26T21:29:56.689622+0000","last_fullsized":"2024-03-26T21:29:56.689622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.963275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981480+0000","last_change":"2024-03-26T21:28:38.864935+0000","last_active":"2024-03-26T21:30:05.981480+0000","last_peered":"2024-03-26T21:30:05.981480+0000","last_clean":"2024-03-26T21:30:05.981480+0000","last_became_active":"2024-03-26T21:28:38.864421+0000","last_became_peered":"2024-03-26T21:28:38.864421+0000","last_unstale":"2024-03-26T21:30:05.981480+0000","last_undegraded":"2024-03-26T21:30:05.981480+0000","last_fullsized":"2024-03-26T21:30:05.981480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:01.423994+0000","last_change":"2024-03-26T21:30:01.424034+0000","last_active":"2024-03-26T21:30:01.423994+0000","last_peered":"2024-03-26T21:30:01.423994+0000","last_clean":"2024-03-26T21:30:01.423994+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:30:01.423994+0000","last_undegraded":"2024-03-26T21:30:01.423994+0000","last_fullsized":"2024-03-26T21:30:01.423994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.423920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.423920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.782127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689657+0000","last_change":"2024-03-26T21:28:37.142935+0000","last_active":"2024-03-26T21:29:56.689657+0000","last_peered":"2024-03-26T21:29:56.689657+0000","last_clean":"2024-03-26T21:29:56.689657+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:29:56.689657+0000","last_undegraded":"2024-03-26T21:29:56.689657+0000","last_fullsized":"2024-03-26T21:29:56.689657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.499596+0000","last_change":"2024-03-26T21:28:40.889084+0000","last_active":"2024-03-26T21:29:56.499596+0000","last_peered":"2024-03-26T21:29:56.499596+0000","last_clean":"2024-03-26T21:29:56.499596+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:29:56.499596+0000","last_undegraded":"2024-03-26T21:29:56.499596+0000","last_fullsized":"2024-03-26T21:29:56.499596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:02.980105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:04.703791+0000","last_change":"2024-03-26T21:30:04.703815+0000","last_active":"2024-03-26T21:30:04.703791+0000","last_peered":"2024-03-26T21:30:04.703791+0000","last_clean":"2024-03-26T21:30:04.703791+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:30:04.703791+0000","last_undegraded":"2024-03-26T21:30:04.703791+0000","last_fullsized":"2024-03-26T21:30:04.703791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.703726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.703726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.302776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981373+0000","last_change":"2024-03-26T21:29:55.577602+0000","last_active":"2024-03-26T21:30:05.981373+0000","last_peered":"2024-03-26T21:30:05.981373+0000","last_clean":"2024-03-26T21:30:05.981373+0000","last_became_active":"2024-03-26T21:28:34.848518+0000","last_became_peered":"2024-03-26T21:28:34.848518+0000","last_unstale":"2024-03-26T21:30:05.981373+0000","last_undegraded":"2024-03-26T21:30:05.981373+0000","last_fullsized":"2024-03-26T21:30:05.981373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.145785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986230+0000","last_change":"2024-03-26T21:28:36.841238+0000","last_active":"2024-03-26T21:30:05.986230+0000","last_peered":"2024-03-26T21:30:05.986230+0000","last_clean":"2024-03-26T21:30:05.986230+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:30:05.986230+0000","last_undegraded":"2024-03-26T21:30:05.986230+0000","last_fullsized":"2024-03-26T21:30:05.986230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689372+0000","last_change":"2024-03-26T21:28:40.891718+0000","last_active":"2024-03-26T21:29:56.689372+0000","last_peered":"2024-03-26T21:29:56.689372+0000","last_clean":"2024-03-26T21:29:56.689372+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:29:56.689372+0000","last_undegraded":"2024-03-26T21:29:56.689372+0000","last_fullsized":"2024-03-26T21:29:56.689372+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986078+0000","last_change":"2024-03-26T21:30:04.095313+0000","last_active":"2024-03-26T21:30:05.986078+0000","last_peered":"2024-03-26T21:30:05.986078+0000","last_clean":"2024-03-26T21:30:05.986078+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:30:05.986078+0000","last_undegraded":"2024-03-26T21:30:05.986078+0000","last_fullsized":"2024-03-26T21:30:05.986078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.095208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.095208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.418775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501177+0000","last_change":"2024-03-26T21:28:34.844448+0000","last_active":"2024-03-26T21:29:56.501177+0000","last_peered":"2024-03-26T21:29:56.501177+0000","last_clean":"2024-03-26T21:29:56.501177+0000","last_became_active":"2024-03-26T21:28:34.842160+0000","last_became_peered":"2024-03-26T21:28:34.842160+0000","last_unstale":"2024-03-26T21:29:56.501177+0000","last_undegraded":"2024-03-26T21:29:56.501177+0000","last_fullsized":"2024-03-26T21:29:56.501177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.941135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500836+0000","last_change":"2024-03-26T21:28:36.846742+0000","last_active":"2024-03-26T21:29:56.500836+0000","last_peered":"2024-03-26T21:29:56.500836+0000","last_clean":"2024-03-26T21:29:56.500836+0000","last_became_active":"2024-03-26T21:28:36.846289+0000","last_became_peered":"2024-03-26T21:28:36.846289+0000","last_unstale":"2024-03-26T21:29:56.500836+0000","last_undegraded":"2024-03-26T21:29:56.500836+0000","last_fullsized":"2024-03-26T21:29:56.500836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:59.376267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:04.667792+0000","last_change":"2024-03-26T21:30:04.667815+0000","last_active":"2024-03-26T21:30:04.667792+0000","last_peered":"2024-03-26T21:30:04.667792+0000","last_clean":"2024-03-26T21:30:04.667792+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:30:04.667792+0000","last_undegraded":"2024-03-26T21:30:04.667792+0000","last_fullsized":"2024-03-26T21:30:04.667792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:30:04.667736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.667736+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.712159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.901803+0000","last_change":"2024-03-26T21:29:47.419090+0000","last_active":"2024-03-26T21:29:59.901803+0000","last_peered":"2024-03-26T21:29:59.901803+0000","last_clean":"2024-03-26T21:29:59.901803+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:29:59.901803+0000","last_undegraded":"2024-03-26T21:29:59.901803+0000","last_fullsized":"2024-03-26T21:29:59.901803+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:29:47.418949+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.219762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981168+0000","last_change":"2024-03-26T21:29:39.550921+0000","last_active":"2024-03-26T21:30:05.981168+0000","last_peered":"2024-03-26T21:30:05.981168+0000","last_clean":"2024-03-26T21:30:05.981168+0000","last_became_active":"2024-03-26T21:28:34.839491+0000","last_became_peered":"2024-03-26T21:28:34.839491+0000","last_unstale":"2024-03-26T21:30:05.981168+0000","last_undegraded":"2024-03-26T21:30:05.981168+0000","last_fullsized":"2024-03-26T21:30:05.981168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.089443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690253+0000","last_change":"2024-03-26T21:29:55.733996+0000","last_active":"2024-03-26T21:29:56.690253+0000","last_peered":"2024-03-26T21:29:56.690253+0000","last_clean":"2024-03-26T21:29:56.690253+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:29:56.690253+0000","last_undegraded":"2024-03-26T21:29:56.690253+0000","last_fullsized":"2024-03-26T21:29:56.690253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:29:55.733905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.733905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.846036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501430+0000","last_change":"2024-03-26T21:28:40.881531+0000","last_active":"2024-03-26T21:29:56.501430+0000","last_peered":"2024-03-26T21:29:56.501430+0000","last_clean":"2024-03-26T21:29:56.501430+0000","last_became_active":"2024-03-26T21:28:40.881234+0000","last_became_peered":"2024-03-26T21:28:40.881234+0000","last_unstale":"2024-03-26T21:29:56.501430+0000","last_undegraded":"2024-03-26T21:29:56.501430+0000","last_fullsized":"2024-03-26T21:29:56.501430+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.460159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.901747+0000","last_change":"2024-03-26T21:28:38.864606+0000","last_active":"2024-03-26T21:29:59.901747+0000","last_peered":"2024-03-26T21:29:59.901747+0000","last_clean":"2024-03-26T21:29:59.901747+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:29:59.901747+0000","last_undegraded":"2024-03-26T21:29:59.901747+0000","last_fullsized":"2024-03-26T21:29:59.901747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.499914+0000","last_change":"2024-03-26T21:28:34.841082+0000","last_active":"2024-03-26T21:29:56.499914+0000","last_peered":"2024-03-26T21:29:56.499914+0000","last_clean":"2024-03-26T21:29:56.499914+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:29:56.499914+0000","last_undegraded":"2024-03-26T21:29:56.499914+0000","last_fullsized":"2024-03-26T21:29:56.499914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.847025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:02.461393+0000","last_change":"2024-03-26T21:30:02.461420+0000","last_active":"2024-03-26T21:30:02.461393+0000","last_peered":"2024-03-26T21:30:02.461393+0000","last_clean":"2024-03-26T21:30:02.461393+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:30:02.461393+0000","last_undegraded":"2024-03-26T21:30:02.461393+0000","last_fullsized":"2024-03-26T21:30:02.461393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:30:02.461332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.461332+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.159192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980841+0000","last_change":"2024-03-26T21:30:05.958764+0000","last_active":"2024-03-26T21:30:05.980841+0000","last_peered":"2024-03-26T21:30:05.980841+0000","last_clean":"2024-03-26T21:30:05.980841+0000","last_became_active":"2024-03-26T21:28:40.884963+0000","last_became_peered":"2024-03-26T21:28:40.884963+0000","last_unstale":"2024-03-26T21:30:05.980841+0000","last_undegraded":"2024-03-26T21:30:05.980841+0000","last_fullsized":"2024-03-26T21:30:05.980841+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.958658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.958658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.251328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690684+0000","last_change":"2024-03-26T21:28:38.864418+0000","last_active":"2024-03-26T21:29:56.690684+0000","last_peered":"2024-03-26T21:29:56.690684+0000","last_clean":"2024-03-26T21:29:56.690684+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:29:56.690684+0000","last_undegraded":"2024-03-26T21:29:56.690684+0000","last_fullsized":"2024-03-26T21:29:56.690684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980943+0000","last_change":"2024-03-26T21:28:34.843600+0000","last_active":"2024-03-26T21:30:05.980943+0000","last_peered":"2024-03-26T21:30:05.980943+0000","last_clean":"2024-03-26T21:30:05.980943+0000","last_became_active":"2024-03-26T21:28:34.843175+0000","last_became_peered":"2024-03-26T21:28:34.843175+0000","last_unstale":"2024-03-26T21:30:05.980943+0000","last_undegraded":"2024-03-26T21:30:05.980943+0000","last_fullsized":"2024-03-26T21:30:05.980943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:00.719863+0000","last_change":"2024-03-26T21:30:00.719906+0000","last_active":"2024-03-26T21:30:00.719863+0000","last_peered":"2024-03-26T21:30:00.719863+0000","last_clean":"2024-03-26T21:30:00.719863+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:30:00.719863+0000","last_undegraded":"2024-03-26T21:30:00.719863+0000","last_fullsized":"2024-03-26T21:30:00.719863+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:30:00.719791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:00.719791+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.215539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981300+0000","last_change":"2024-03-26T21:28:41.115111+0000","last_active":"2024-03-26T21:30:05.981300+0000","last_peered":"2024-03-26T21:30:05.981300+0000","last_clean":"2024-03-26T21:30:05.981300+0000","last_became_active":"2024-03-26T21:28:41.114831+0000","last_became_peered":"2024-03-26T21:28:41.114831+0000","last_unstale":"2024-03-26T21:30:05.981300+0000","last_undegraded":"2024-03-26T21:30:05.981300+0000","last_fullsized":"2024-03-26T21:30:05.981300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:03.444609+0000","last_change":"2024-03-26T21:30:03.444638+0000","last_active":"2024-03-26T21:30:03.444609+0000","last_peered":"2024-03-26T21:30:03.444609+0000","last_clean":"2024-03-26T21:30:03.444609+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:30:03.444609+0000","last_undegraded":"2024-03-26T21:30:03.444609+0000","last_fullsized":"2024-03-26T21:30:03.444609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.444530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.444530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.858717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.496525+0000","last_change":"2024-03-26T21:29:53.937502+0000","last_active":"2024-03-26T21:30:06.496525+0000","last_peered":"2024-03-26T21:30:06.496525+0000","last_clean":"2024-03-26T21:30:06.496525+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:30:06.496525+0000","last_undegraded":"2024-03-26T21:30:06.496525+0000","last_fullsized":"2024-03-26T21:30:06.496525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:53.937386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:53.937386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.079593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690605+0000","last_change":"2024-03-26T21:28:59.687904+0000","last_active":"2024-03-26T21:29:56.690605+0000","last_peered":"2024-03-26T21:29:56.690605+0000","last_clean":"2024-03-26T21:29:56.690605+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:29:56.690605+0000","last_undegraded":"2024-03-26T21:29:56.690605+0000","last_fullsized":"2024-03-26T21:29:56.690605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.141905+0000","last_change":"2024-03-26T21:28:41.114110+0000","last_active":"2024-03-26T21:29:57.141905+0000","last_peered":"2024-03-26T21:29:57.141905+0000","last_clean":"2024-03-26T21:29:57.141905+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:29:57.141905+0000","last_undegraded":"2024-03-26T21:29:57.141905+0000","last_fullsized":"2024-03-26T21:29:57.141905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981555+0000","last_change":"2024-03-26T21:28:34.843004+0000","last_active":"2024-03-26T21:30:05.981555+0000","last_peered":"2024-03-26T21:30:05.981555+0000","last_clean":"2024-03-26T21:30:05.981555+0000","last_became_active":"2024-03-26T21:28:34.842712+0000","last_became_peered":"2024-03-26T21:28:34.842712+0000","last_unstale":"2024-03-26T21:30:05.981555+0000","last_undegraded":"2024-03-26T21:30:05.981555+0000","last_fullsized":"2024-03-26T21:30:05.981555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.141598+0000","last_change":"2024-03-26T21:28:37.141762+0000","last_active":"2024-03-26T21:29:57.141598+0000","last_peered":"2024-03-26T21:29:57.141598+0000","last_clean":"2024-03-26T21:29:57.141598+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:57.141598+0000","last_undegraded":"2024-03-26T21:29:57.141598+0000","last_fullsized":"2024-03-26T21:29:57.141598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981140+0000","last_change":"2024-03-26T21:28:38.874954+0000","last_active":"2024-03-26T21:30:05.981140+0000","last_peered":"2024-03-26T21:30:05.981140+0000","last_clean":"2024-03-26T21:30:05.981140+0000","last_became_active":"2024-03-26T21:28:38.874646+0000","last_became_peered":"2024-03-26T21:28:38.874646+0000","last_unstale":"2024-03-26T21:30:05.981140+0000","last_undegraded":"2024-03-26T21:30:05.981140+0000","last_fullsized":"2024-03-26T21:30:05.981140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689694+0000","last_change":"2024-03-26T21:28:40.888897+0000","last_active":"2024-03-26T21:29:56.689694+0000","last_peered":"2024-03-26T21:29:56.689694+0000","last_clean":"2024-03-26T21:29:56.689694+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:29:56.689694+0000","last_undegraded":"2024-03-26T21:29:56.689694+0000","last_fullsized":"2024-03-26T21:29:56.689694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:08.336674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981525+0000","last_change":"2024-03-26T21:29:44.612815+0000","last_active":"2024-03-26T21:30:05.981525+0000","last_peered":"2024-03-26T21:30:05.981525+0000","last_clean":"2024-03-26T21:30:05.981525+0000","last_became_active":"2024-03-26T21:28:34.848110+0000","last_became_peered":"2024-03-26T21:28:34.848110+0000","last_unstale":"2024-03-26T21:30:05.981525+0000","last_undegraded":"2024-03-26T21:30:05.981525+0000","last_fullsized":"2024-03-26T21:30:05.981525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.133060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500601+0000","last_change":"2024-03-26T21:29:42.608574+0000","last_active":"2024-03-26T21:29:56.500601+0000","last_peered":"2024-03-26T21:29:56.500601+0000","last_clean":"2024-03-26T21:29:56.500601+0000","last_became_active":"2024-03-26T21:28:36.847905+0000","last_became_peered":"2024-03-26T21:28:36.847905+0000","last_unstale":"2024-03-26T21:29:56.500601+0000","last_undegraded":"2024-03-26T21:29:56.500601+0000","last_fullsized":"2024-03-26T21:29:56.500601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.811581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980401+0000","last_change":"2024-03-26T21:30:02.905489+0000","last_active":"2024-03-26T21:30:05.980401+0000","last_peered":"2024-03-26T21:30:05.980401+0000","last_clean":"2024-03-26T21:30:05.980401+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:30:05.980401+0000","last_undegraded":"2024-03-26T21:30:05.980401+0000","last_fullsized":"2024-03-26T21:30:05.980401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:02.905408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.905408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.664056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980719+0000","last_change":"2024-03-26T21:28:40.887205+0000","last_active":"2024-03-26T21:30:05.980719+0000","last_peered":"2024-03-26T21:30:05.980719+0000","last_clean":"2024-03-26T21:30:05.980719+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:30:05.980719+0000","last_undegraded":"2024-03-26T21:30:05.980719+0000","last_fullsized":"2024-03-26T21:30:05.980719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980632+0000","last_change":"2024-03-26T21:30:03.938919+0000","last_active":"2024-03-26T21:30:05.980632+0000","last_peered":"2024-03-26T21:30:05.980632+0000","last_clean":"2024-03-26T21:30:05.980632+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:30:05.980632+0000","last_undegraded":"2024-03-26T21:30:05.980632+0000","last_fullsized":"2024-03-26T21:30:05.980632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.938836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.938836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.038793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981503+0000","last_change":"2024-03-26T21:29:48.625718+0000","last_active":"2024-03-26T21:30:05.981503+0000","last_peered":"2024-03-26T21:30:05.981503+0000","last_clean":"2024-03-26T21:30:05.981503+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:30:05.981503+0000","last_undegraded":"2024-03-26T21:30:05.981503+0000","last_fullsized":"2024-03-26T21:30:05.981503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.625595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.625595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.303805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497154+0000","last_change":"2024-03-26T21:29:58.792883+0000","last_active":"2024-03-26T21:30:06.497154+0000","last_peered":"2024-03-26T21:30:06.497154+0000","last_clean":"2024-03-26T21:30:06.497154+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:30:06.497154+0000","last_undegraded":"2024-03-26T21:30:06.497154+0000","last_fullsized":"2024-03-26T21:30:06.497154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:58.792783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.792783+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:02.267914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.499686+0000","last_change":"2024-03-26T21:28:40.889290+0000","last_active":"2024-03-26T21:29:56.499686+0000","last_peered":"2024-03-26T21:29:56.499686+0000","last_clean":"2024-03-26T21:29:56.499686+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:29:56.499686+0000","last_undegraded":"2024-03-26T21:29:56.499686+0000","last_fullsized":"2024-03-26T21:29:56.499686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980549+0000","last_change":"2024-03-26T21:29:48.738742+0000","last_active":"2024-03-26T21:30:05.980549+0000","last_peered":"2024-03-26T21:30:05.980549+0000","last_clean":"2024-03-26T21:30:05.980549+0000","last_became_active":"2024-03-26T21:28:38.866104+0000","last_became_peered":"2024-03-26T21:28:38.866104+0000","last_unstale":"2024-03-26T21:30:05.980549+0000","last_undegraded":"2024-03-26T21:30:05.980549+0000","last_fullsized":"2024-03-26T21:30:05.980549+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.530253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142172+0000","last_change":"2024-03-26T21:28:34.841686+0000","last_active":"2024-03-26T21:29:57.142172+0000","last_peered":"2024-03-26T21:29:57.142172+0000","last_clean":"2024-03-26T21:29:57.142172+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:29:57.142172+0000","last_undegraded":"2024-03-26T21:29:57.142172+0000","last_fullsized":"2024-03-26T21:29:57.142172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.267735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501505+0000","last_change":"2024-03-26T21:28:36.845693+0000","last_active":"2024-03-26T21:29:56.501505+0000","last_peered":"2024-03-26T21:29:56.501505+0000","last_clean":"2024-03-26T21:29:56.501505+0000","last_became_active":"2024-03-26T21:28:36.845358+0000","last_became_peered":"2024-03-26T21:28:36.845358+0000","last_unstale":"2024-03-26T21:29:56.501505+0000","last_undegraded":"2024-03-26T21:29:56.501505+0000","last_fullsized":"2024-03-26T21:29:56.501505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.265401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690252+0000","last_change":"2024-03-26T21:29:42.726419+0000","last_active":"2024-03-26T21:29:56.690252+0000","last_peered":"2024-03-26T21:29:56.690252+0000","last_clean":"2024-03-26T21:29:56.690252+0000","last_became_active":"2024-03-26T21:28:40.884694+0000","last_became_peered":"2024-03-26T21:28:40.884694+0000","last_unstale":"2024-03-26T21:29:56.690252+0000","last_undegraded":"2024-03-26T21:29:56.690252+0000","last_fullsized":"2024-03-26T21:29:56.690252+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.138620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.141730+0000","last_change":"2024-03-26T21:28:38.867924+0000","last_active":"2024-03-26T21:29:57.141730+0000","last_peered":"2024-03-26T21:29:57.141730+0000","last_clean":"2024-03-26T21:29:57.141730+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:29:57.141730+0000","last_undegraded":"2024-03-26T21:29:57.141730+0000","last_fullsized":"2024-03-26T21:29:57.141730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500562+0000","last_change":"2024-03-26T21:28:34.842465+0000","last_active":"2024-03-26T21:29:56.500562+0000","last_peered":"2024-03-26T21:29:56.500562+0000","last_clean":"2024-03-26T21:29:56.500562+0000","last_became_active":"2024-03-26T21:28:34.842130+0000","last_became_peered":"2024-03-26T21:28:34.842130+0000","last_unstale":"2024-03-26T21:29:56.500562+0000","last_undegraded":"2024-03-26T21:29:56.500562+0000","last_fullsized":"2024-03-26T21:29:56.500562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.625559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.902137+0000","last_change":"2024-03-26T21:29:56.452291+0000","last_active":"2024-03-26T21:29:59.902137+0000","last_peered":"2024-03-26T21:29:59.902137+0000","last_clean":"2024-03-26T21:29:59.902137+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:29:59.902137+0000","last_undegraded":"2024-03-26T21:29:59.902137+0000","last_fullsized":"2024-03-26T21:29:59.902137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:29:56.452170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.452170+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:14.545743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500321+0000","last_change":"2024-03-26T21:28:38.873506+0000","last_active":"2024-03-26T21:29:56.500321+0000","last_peered":"2024-03-26T21:29:56.500321+0000","last_clean":"2024-03-26T21:29:56.500321+0000","last_became_active":"2024-03-26T21:28:38.873196+0000","last_became_peered":"2024-03-26T21:28:38.873196+0000","last_unstale":"2024-03-26T21:29:56.500321+0000","last_undegraded":"2024-03-26T21:29:56.500321+0000","last_fullsized":"2024-03-26T21:29:56.500321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981604+0000","last_change":"2024-03-26T21:28:41.114073+0000","last_active":"2024-03-26T21:30:05.981604+0000","last_peered":"2024-03-26T21:30:05.981604+0000","last_clean":"2024-03-26T21:30:05.981604+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:30:05.981604+0000","last_undegraded":"2024-03-26T21:30:05.981604+0000","last_fullsized":"2024-03-26T21:30:05.981604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985798+0000","last_change":"2024-03-26T21:29:36.249432+0000","last_active":"2024-03-26T21:30:05.985798+0000","last_peered":"2024-03-26T21:30:05.985798+0000","last_clean":"2024-03-26T21:30:05.985798+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:30:05.985798+0000","last_undegraded":"2024-03-26T21:30:05.985798+0000","last_fullsized":"2024-03-26T21:30:05.985798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:36.249302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.249302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:39.117721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980486+0000","last_change":"2024-03-26T21:29:41.695430+0000","last_active":"2024-03-26T21:30:05.980486+0000","last_peered":"2024-03-26T21:30:05.980486+0000","last_clean":"2024-03-26T21:30:05.980486+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:30:05.980486+0000","last_undegraded":"2024-03-26T21:30:05.980486+0000","last_fullsized":"2024-03-26T21:30:05.980486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.695322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.695322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.842596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986561+0000","last_change":"2024-03-26T21:29:55.054708+0000","last_active":"2024-03-26T21:30:05.986561+0000","last_peered":"2024-03-26T21:30:05.986561+0000","last_clean":"2024-03-26T21:30:05.986561+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:30:05.986561+0000","last_undegraded":"2024-03-26T21:30:05.986561+0000","last_fullsized":"2024-03-26T21:30:05.986561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.054601+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.383564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980741+0000","last_change":"2024-03-26T21:28:38.874086+0000","last_active":"2024-03-26T21:30:05.980741+0000","last_peered":"2024-03-26T21:30:05.980741+0000","last_clean":"2024-03-26T21:30:05.980741+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:30:05.980741+0000","last_undegraded":"2024-03-26T21:30:05.980741+0000","last_fullsized":"2024-03-26T21:30:05.980741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142318+0000","last_change":"2024-03-26T21:28:40.887390+0000","last_active":"2024-03-26T21:29:57.142318+0000","last_peered":"2024-03-26T21:29:57.142318+0000","last_clean":"2024-03-26T21:29:57.142318+0000","last_became_active":"2024-03-26T21:28:40.886965+0000","last_became_peered":"2024-03-26T21:28:40.886965+0000","last_unstale":"2024-03-26T21:29:57.142318+0000","last_undegraded":"2024-03-26T21:29:57.142318+0000","last_fullsized":"2024-03-26T21:29:57.142318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":310,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980852+0000","last_change":"2024-03-26T21:28:42.275445+0000","last_active":"2024-03-26T21:30:05.980852+0000","last_peered":"2024-03-26T21:30:05.980852+0000","last_clean":"2024-03-26T21:30:05.980852+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:30:05.980852+0000","last_undegraded":"2024-03-26T21:30:05.980852+0000","last_fullsized":"2024-03-26T21:30:05.980852+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T21:28:42.275350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:28:42.275350+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689656+0000","last_change":"2024-03-26T21:29:38.690870+0000","last_active":"2024-03-26T21:29:56.689656+0000","last_peered":"2024-03-26T21:29:56.689656+0000","last_clean":"2024-03-26T21:29:56.689656+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:29:56.689656+0000","last_undegraded":"2024-03-26T21:29:56.689656+0000","last_fullsized":"2024-03-26T21:29:56.689656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.690779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.690779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.230074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"88'24","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497472+0000","last_change":"2024-03-26T21:29:55.903181+0000","last_active":"2024-03-26T21:30:06.497472+0000","last_peered":"2024-03-26T21:30:06.497472+0000","last_clean":"2024-03-26T21:30:06.497472+0000","last_became_active":"2024-03-26T21:28:37.140774+0000","last_became_peered":"2024-03-26T21:28:37.140774+0000","last_unstale":"2024-03-26T21:30:06.497472+0000","last_undegraded":"2024-03-26T21:30:06.497472+0000","last_fullsized":"2024-03-26T21:30:06.497472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:13.673385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981733+0000","last_change":"2024-03-26T21:28:38.868716+0000","last_active":"2024-03-26T21:30:05.981733+0000","last_peered":"2024-03-26T21:30:05.981733+0000","last_clean":"2024-03-26T21:30:05.981733+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:30:05.981733+0000","last_undegraded":"2024-03-26T21:30:05.981733+0000","last_fullsized":"2024-03-26T21:30:05.981733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986306+0000","last_change":"2024-03-26T21:28:40.887424+0000","last_active":"2024-03-26T21:30:05.986306+0000","last_peered":"2024-03-26T21:30:05.986306+0000","last_clean":"2024-03-26T21:30:05.986306+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:30:05.986306+0000","last_undegraded":"2024-03-26T21:30:05.986306+0000","last_fullsized":"2024-03-26T21:30:05.986306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.499836+0000","last_change":"2024-03-26T21:28:34.843666+0000","last_active":"2024-03-26T21:29:56.499836+0000","last_peered":"2024-03-26T21:29:56.499836+0000","last_clean":"2024-03-26T21:29:56.499836+0000","last_became_active":"2024-03-26T21:28:34.843392+0000","last_became_peered":"2024-03-26T21:28:34.843392+0000","last_unstale":"2024-03-26T21:29:56.499836+0000","last_undegraded":"2024-03-26T21:29:56.499836+0000","last_fullsized":"2024-03-26T21:29:56.499836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":154,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980794+0000","last_change":"2024-03-26T21:28:38.848730+0000","last_active":"2024-03-26T21:30:05.980794+0000","last_peered":"2024-03-26T21:30:05.980794+0000","last_clean":"2024-03-26T21:30:05.980794+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:30:05.980794+0000","last_undegraded":"2024-03-26T21:30:05.980794+0000","last_fullsized":"2024-03-26T21:30:05.980794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501200+0000","last_change":"2024-03-26T21:29:38.920799+0000","last_active":"2024-03-26T21:29:56.501200+0000","last_peered":"2024-03-26T21:29:56.501200+0000","last_clean":"2024-03-26T21:29:56.501200+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:29:56.501200+0000","last_undegraded":"2024-03-26T21:29:56.501200+0000","last_fullsized":"2024-03-26T21:29:56.501200+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:38.920701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.920701+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.589043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142590+0000","last_change":"2024-03-26T21:28:38.870606+0000","last_active":"2024-03-26T21:29:57.142590+0000","last_peered":"2024-03-26T21:29:57.142590+0000","last_clean":"2024-03-26T21:29:57.142590+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:29:57.142590+0000","last_undegraded":"2024-03-26T21:29:57.142590+0000","last_fullsized":"2024-03-26T21:29:57.142590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:07.288276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985698+0000","last_change":"2024-03-26T21:30:03.078966+0000","last_active":"2024-03-26T21:30:05.985698+0000","last_peered":"2024-03-26T21:30:05.985698+0000","last_clean":"2024-03-26T21:30:05.985698+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:30:05.985698+0000","last_undegraded":"2024-03-26T21:30:05.985698+0000","last_fullsized":"2024-03-26T21:30:05.985698+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.078885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.078885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:20.097118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985977+0000","last_change":"2024-03-26T21:29:42.167843+0000","last_active":"2024-03-26T21:30:05.985977+0000","last_peered":"2024-03-26T21:30:05.985977+0000","last_clean":"2024-03-26T21:30:05.985977+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:30:05.985977+0000","last_undegraded":"2024-03-26T21:30:05.985977+0000","last_fullsized":"2024-03-26T21:30:05.985977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.167719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:45.815629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501397+0000","last_change":"2024-03-26T21:29:51.863137+0000","last_active":"2024-03-26T21:29:56.501397+0000","last_peered":"2024-03-26T21:29:56.501397+0000","last_clean":"2024-03-26T21:29:56.501397+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:29:56.501397+0000","last_undegraded":"2024-03-26T21:29:56.501397+0000","last_fullsized":"2024-03-26T21:29:56.501397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:51.862960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.862960+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:12.041689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497165+0000","last_change":"2024-03-26T21:29:52.956781+0000","last_active":"2024-03-26T21:30:06.497165+0000","last_peered":"2024-03-26T21:30:06.497165+0000","last_clean":"2024-03-26T21:30:06.497165+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:30:06.497165+0000","last_undegraded":"2024-03-26T21:30:06.497165+0000","last_fullsized":"2024-03-26T21:30:06.497165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.956669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.956669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:54.531858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.499597+0000","last_change":"2024-03-26T21:28:40.890739+0000","last_active":"2024-03-26T21:29:56.499597+0000","last_peered":"2024-03-26T21:29:56.499597+0000","last_clean":"2024-03-26T21:29:56.499597+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:29:56.499597+0000","last_undegraded":"2024-03-26T21:29:56.499597+0000","last_fullsized":"2024-03-26T21:29:56.499597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142223+0000","last_change":"2024-03-26T21:29:38.189988+0000","last_active":"2024-03-26T21:29:57.142223+0000","last_peered":"2024-03-26T21:29:57.142223+0000","last_clean":"2024-03-26T21:29:57.142223+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:29:57.142223+0000","last_undegraded":"2024-03-26T21:29:57.142223+0000","last_fullsized":"2024-03-26T21:29:57.142223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.189877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.189877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:38.248593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986154+0000","last_change":"2024-03-26T21:30:02.076222+0000","last_active":"2024-03-26T21:30:05.986154+0000","last_peered":"2024-03-26T21:30:05.986154+0000","last_clean":"2024-03-26T21:30:05.986154+0000","last_became_active":"2024-03-26T21:28:36.841759+0000","last_became_peered":"2024-03-26T21:28:36.841759+0000","last_unstale":"2024-03-26T21:30:05.986154+0000","last_undegraded":"2024-03-26T21:30:05.986154+0000","last_fullsized":"2024-03-26T21:30:05.986154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:02.076064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.076064+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:14.477601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.901737+0000","last_change":"2024-03-26T21:28:38.869750+0000","last_active":"2024-03-26T21:29:59.901737+0000","last_peered":"2024-03-26T21:29:59.901737+0000","last_clean":"2024-03-26T21:29:59.901737+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:29:59.901737+0000","last_undegraded":"2024-03-26T21:29:59.901737+0000","last_fullsized":"2024-03-26T21:29:59.901737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981576+0000","last_change":"2024-03-26T21:28:41.115853+0000","last_active":"2024-03-26T21:30:05.981576+0000","last_peered":"2024-03-26T21:30:05.981576+0000","last_clean":"2024-03-26T21:30:05.981576+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:30:05.981576+0000","last_undegraded":"2024-03-26T21:30:05.981576+0000","last_fullsized":"2024-03-26T21:30:05.981576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501299+0000","last_change":"2024-03-26T21:28:34.846498+0000","last_active":"2024-03-26T21:29:56.501299+0000","last_peered":"2024-03-26T21:29:56.501299+0000","last_clean":"2024-03-26T21:29:56.501299+0000","last_became_active":"2024-03-26T21:28:34.846219+0000","last_became_peered":"2024-03-26T21:28:34.846219+0000","last_unstale":"2024-03-26T21:29:56.501299+0000","last_undegraded":"2024-03-26T21:29:56.501299+0000","last_fullsized":"2024-03-26T21:29:56.501299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500519+0000","last_change":"2024-03-26T21:28:36.840120+0000","last_active":"2024-03-26T21:29:56.500519+0000","last_peered":"2024-03-26T21:29:56.500519+0000","last_clean":"2024-03-26T21:29:56.500519+0000","last_became_active":"2024-03-26T21:28:36.839780+0000","last_became_peered":"2024-03-26T21:28:36.839780+0000","last_unstale":"2024-03-26T21:29:56.500519+0000","last_undegraded":"2024-03-26T21:29:56.500519+0000","last_fullsized":"2024-03-26T21:29:56.500519+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.901761+0000","last_change":"2024-03-26T21:29:59.694526+0000","last_active":"2024-03-26T21:29:59.901761+0000","last_peered":"2024-03-26T21:29:59.901761+0000","last_clean":"2024-03-26T21:29:59.901761+0000","last_became_active":"2024-03-26T21:28:38.864364+0000","last_became_peered":"2024-03-26T21:28:38.864364+0000","last_unstale":"2024-03-26T21:29:59.901761+0000","last_undegraded":"2024-03-26T21:29:59.901761+0000","last_fullsized":"2024-03-26T21:29:59.901761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:29:59.694434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:59.694434+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.427100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500386+0000","last_change":"2024-03-26T21:29:48.632454+0000","last_active":"2024-03-26T21:29:56.500386+0000","last_peered":"2024-03-26T21:29:56.500386+0000","last_clean":"2024-03-26T21:29:56.500386+0000","last_became_active":"2024-03-26T21:28:40.886497+0000","last_became_peered":"2024-03-26T21:28:40.886497+0000","last_unstale":"2024-03-26T21:29:56.500386+0000","last_undegraded":"2024-03-26T21:29:56.500386+0000","last_fullsized":"2024-03-26T21:29:56.500386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.661591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690945+0000","last_change":"2024-03-26T21:29:56.690968+0000","last_active":"2024-03-26T21:29:56.690945+0000","last_peered":"2024-03-26T21:29:56.690945+0000","last_clean":"2024-03-26T21:29:56.690945+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:29:56.690945+0000","last_undegraded":"2024-03-26T21:29:56.690945+0000","last_fullsized":"2024-03-26T21:29:56.690945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.690884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.690884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.513440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689400+0000","last_change":"2024-03-26T21:28:37.143706+0000","last_active":"2024-03-26T21:29:56.689400+0000","last_peered":"2024-03-26T21:29:56.689400+0000","last_clean":"2024-03-26T21:29:56.689400+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:29:56.689400+0000","last_undegraded":"2024-03-26T21:29:56.689400+0000","last_fullsized":"2024-03-26T21:29:56.689400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500250+0000","last_change":"2024-03-26T21:28:38.863832+0000","last_active":"2024-03-26T21:29:56.500250+0000","last_peered":"2024-03-26T21:29:56.500250+0000","last_clean":"2024-03-26T21:29:56.500250+0000","last_became_active":"2024-03-26T21:28:38.863503+0000","last_became_peered":"2024-03-26T21:28:38.863503+0000","last_unstale":"2024-03-26T21:29:56.500250+0000","last_undegraded":"2024-03-26T21:29:56.500250+0000","last_fullsized":"2024-03-26T21:29:56.500250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:01.090750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:04.491855+0000","last_change":"2024-03-26T21:30:04.491878+0000","last_active":"2024-03-26T21:30:04.491855+0000","last_peered":"2024-03-26T21:30:04.491855+0000","last_clean":"2024-03-26T21:30:04.491855+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:30:04.491855+0000","last_undegraded":"2024-03-26T21:30:04.491855+0000","last_fullsized":"2024-03-26T21:30:04.491855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.491791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.670339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:59.901961+0000","last_change":"2024-03-26T21:28:38.871742+0000","last_active":"2024-03-26T21:29:59.901961+0000","last_peered":"2024-03-26T21:29:59.901961+0000","last_clean":"2024-03-26T21:29:59.901961+0000","last_became_active":"2024-03-26T21:28:38.871425+0000","last_became_peered":"2024-03-26T21:28:38.871425+0000","last_unstale":"2024-03-26T21:29:59.901961+0000","last_undegraded":"2024-03-26T21:29:59.901961+0000","last_fullsized":"2024-03-26T21:29:59.901961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689339+0000","last_change":"2024-03-26T21:29:41.689438+0000","last_active":"2024-03-26T21:29:56.689339+0000","last_peered":"2024-03-26T21:29:56.689339+0000","last_clean":"2024-03-26T21:29:56.689339+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:29:56.689339+0000","last_undegraded":"2024-03-26T21:29:56.689339+0000","last_fullsized":"2024-03-26T21:29:56.689339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.689332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.689332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.552879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986621+0000","last_change":"2024-03-26T21:28:36.848001+0000","last_active":"2024-03-26T21:30:05.986621+0000","last_peered":"2024-03-26T21:30:05.986621+0000","last_clean":"2024-03-26T21:30:05.986621+0000","last_became_active":"2024-03-26T21:28:36.847705+0000","last_became_peered":"2024-03-26T21:28:36.847705+0000","last_unstale":"2024-03-26T21:30:05.986621+0000","last_undegraded":"2024-03-26T21:30:05.986621+0000","last_fullsized":"2024-03-26T21:30:05.986621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.690045+0000","last_change":"2024-03-26T21:28:41.113828+0000","last_active":"2024-03-26T21:29:56.690045+0000","last_peered":"2024-03-26T21:29:56.690045+0000","last_clean":"2024-03-26T21:29:56.690045+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:29:56.690045+0000","last_undegraded":"2024-03-26T21:29:56.690045+0000","last_fullsized":"2024-03-26T21:29:56.690045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:57.142565+0000","last_change":"2024-03-26T21:28:38.872061+0000","last_active":"2024-03-26T21:29:57.142565+0000","last_peered":"2024-03-26T21:29:57.142565+0000","last_clean":"2024-03-26T21:29:57.142565+0000","last_became_active":"2024-03-26T21:28:38.869931+0000","last_became_peered":"2024-03-26T21:28:38.869931+0000","last_unstale":"2024-03-26T21:29:57.142565+0000","last_undegraded":"2024-03-26T21:29:57.142565+0000","last_fullsized":"2024-03-26T21:29:57.142565+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:06.256868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500284+0000","last_change":"2024-03-26T21:29:50.575715+0000","last_active":"2024-03-26T21:29:56.500284+0000","last_peered":"2024-03-26T21:29:56.500284+0000","last_clean":"2024-03-26T21:29:56.500284+0000","last_became_active":"2024-03-26T21:28:34.842398+0000","last_became_peered":"2024-03-26T21:28:34.842398+0000","last_unstale":"2024-03-26T21:29:56.500284+0000","last_undegraded":"2024-03-26T21:29:56.500284+0000","last_fullsized":"2024-03-26T21:29:56.500284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.357015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:30:01.544815+0000","last_change":"2024-03-26T21:30:01.544846+0000","last_active":"2024-03-26T21:30:01.544815+0000","last_peered":"2024-03-26T21:30:01.544815+0000","last_clean":"2024-03-26T21:30:01.544815+0000","last_became_active":"2024-03-26T21:28:36.841703+0000","last_became_peered":"2024-03-26T21:28:36.841703+0000","last_unstale":"2024-03-26T21:30:01.544815+0000","last_undegraded":"2024-03-26T21:30:01.544815+0000","last_fullsized":"2024-03-26T21:30:01.544815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:04.152930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980913+0000","last_change":"2024-03-26T21:30:04.923415+0000","last_active":"2024-03-26T21:30:05.980913+0000","last_peered":"2024-03-26T21:30:05.980913+0000","last_clean":"2024-03-26T21:30:05.980913+0000","last_became_active":"2024-03-26T21:28:40.884823+0000","last_became_peered":"2024-03-26T21:28:40.884823+0000","last_unstale":"2024-03-26T21:30:05.980913+0000","last_undegraded":"2024-03-26T21:30:05.980913+0000","last_fullsized":"2024-03-26T21:30:05.980913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.923329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.736283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986297+0000","last_change":"2024-03-26T21:29:40.185232+0000","last_active":"2024-03-26T21:30:05.986297+0000","last_peered":"2024-03-26T21:30:05.986297+0000","last_clean":"2024-03-26T21:30:05.986297+0000","last_became_active":"2024-03-26T21:28:38.867110+0000","last_became_peered":"2024-03-26T21:28:38.867110+0000","last_unstale":"2024-03-26T21:30:05.986297+0000","last_undegraded":"2024-03-26T21:30:05.986297+0000","last_fullsized":"2024-03-26T21:30:05.986297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.295284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497012+0000","last_change":"2024-03-26T21:29:39.867915+0000","last_active":"2024-03-26T21:30:06.497012+0000","last_peered":"2024-03-26T21:30:06.497012+0000","last_clean":"2024-03-26T21:30:06.497012+0000","last_became_active":"2024-03-26T21:28:34.846385+0000","last_became_peered":"2024-03-26T21:28:34.846385+0000","last_unstale":"2024-03-26T21:30:06.497012+0000","last_undegraded":"2024-03-26T21:30:06.497012+0000","last_fullsized":"2024-03-26T21:30:06.497012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:51.580336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.501468+0000","last_change":"2024-03-26T21:28:36.842153+0000","last_active":"2024-03-26T21:29:56.501468+0000","last_peered":"2024-03-26T21:29:56.501468+0000","last_clean":"2024-03-26T21:29:56.501468+0000","last_became_active":"2024-03-26T21:28:36.841769+0000","last_became_peered":"2024-03-26T21:28:36.841769+0000","last_unstale":"2024-03-26T21:29:56.501468+0000","last_undegraded":"2024-03-26T21:29:56.501468+0000","last_fullsized":"2024-03-26T21:29:56.501468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:57.653845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:58.537224+0000","last_change":"2024-03-26T21:29:58.537245+0000","last_active":"2024-03-26T21:29:58.537224+0000","last_peered":"2024-03-26T21:29:58.537224+0000","last_clean":"2024-03-26T21:29:58.537224+0000","last_became_active":"2024-03-26T21:28:40.879974+0000","last_became_peered":"2024-03-26T21:28:40.879974+0000","last_unstale":"2024-03-26T21:29:58.537224+0000","last_undegraded":"2024-03-26T21:29:58.537224+0000","last_fullsized":"2024-03-26T21:29:58.537224+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.657997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980667+0000","last_change":"2024-03-26T21:30:01.872594+0000","last_active":"2024-03-26T21:30:05.980667+0000","last_peered":"2024-03-26T21:30:05.980667+0000","last_clean":"2024-03-26T21:30:05.980667+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:30:05.980667+0000","last_undegraded":"2024-03-26T21:30:05.980667+0000","last_fullsized":"2024-03-26T21:30:05.980667+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:01.872464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.872464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.137769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.689867+0000","last_change":"2024-03-26T21:28:34.848313+0000","last_active":"2024-03-26T21:29:56.689867+0000","last_peered":"2024-03-26T21:29:56.689867+0000","last_clean":"2024-03-26T21:29:56.689867+0000","last_became_active":"2024-03-26T21:28:34.848009+0000","last_became_peered":"2024-03-26T21:28:34.848009+0000","last_unstale":"2024-03-26T21:29:56.689867+0000","last_undegraded":"2024-03-26T21:29:56.689867+0000","last_fullsized":"2024-03-26T21:29:56.689867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:29:58.325454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-26T21:29:56.500737+0000","last_change":"2024-03-26T21:29:37.647253+0000","last_active":"2024-03-26T21:29:56.500737+0000","last_peered":"2024-03-26T21:29:56.500737+0000","last_clean":"2024-03-26T21:29:56.500737+0000","last_became_active":"2024-03-26T21:28:36.847902+0000","last_became_peered":"2024-03-26T21:28:36.847902+0000","last_unstale":"2024-03-26T21:29:56.500737+0000","last_undegraded":"2024-03-26T21:29:56.500737+0000","last_fullsized":"2024-03-26T21:29:56.500737+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.419033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070487,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1417216,"data_stored":950466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364830,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716892,"statfs":{"total":95995035648,"available":95966097408,"internally_reserved":0,"allocated":1273856,"data_stored":911437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":733184,"data_stored":322932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32560,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712592,"statfs":{"total":95995035648,"available":95961694208,"internally_reserved":0,"allocated":1417216,"data_stored":949772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":786432,"data_stored":357801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":358590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":827392,"data_stored":359638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":358370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:30:13.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:30:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 6.19 deep-scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 6.19 deep-scrub ok 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 6.c deep-scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[40587]: 6.c deep-scrub ok 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 6.19 deep-scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 6.19 deep-scrub ok 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:30:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:30:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:30:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:30:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 6.c deep-scrub starts 2024-03-26T21:30:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:13 smithi055 ceph-mon[29406]: 6.c deep-scrub ok 2024-03-26T21:30:13.469 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 6.19 deep-scrub starts 2024-03-26T21:30:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 6.19 deep-scrub ok 2024-03-26T21:30:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:30:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:30:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:30:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:30:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 6.c deep-scrub starts 2024-03-26T21:30:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:13 smithi067 ceph-mon[33346]: 6.c deep-scrub ok 2024-03-26T21:30:14.315 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:14 smithi055 bash[76226]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[29406]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[40587]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:30:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:14 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:30:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:14 smithi067 ceph-mon[33346]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:14 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:30:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:14 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:30:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:14 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:30:14.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:14 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:30:14.790 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:14 smithi055 bash[76226]: Writing manifest to image destination 2024-03-26T21:30:15.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-mon[29406]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:15.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-mon[40587]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:15.457 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 podman[76226]: 2024-03-26 21:30:15.104359094 +0000 UTC m=+3.645104761 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T21:30:15.458 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 podman[76226]: 2024-03-26 21:30:15.227265121 +0000 UTC m=+3.768010766 container create 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:15 smithi067 ceph-mon[33346]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:30:16.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:30:16.162 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:30:16.162 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:30:16.162 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:16.162 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 podman[76226]: 2024-03-26 21:30:15.959560067 +0000 UTC m=+4.500305723 container init 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:16.162 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 podman[76226]: 2024-03-26 21:30:15.962859736 +0000 UTC m=+4.503605385 container start 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.967Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T21:30:16.163 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T21:30:16.164 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T21:30:16.165 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.968Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.969Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:15 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a[76424]: ts=2024-03-26T21:30:15.969Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:16 smithi055 bash[76226]: 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 2024-03-26T21:30:16.166 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:30:16 smithi055 systemd[1]: Started Ceph node-exporter.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:16.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:30:16.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:30:16.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:30:16.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:30:16.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:16.739 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:30:16.741 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:30:16.742 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:17.331 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 systemd[1]: Starting Ceph node-exporter.b for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:17.721 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 bash[64603]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T21:30:17.721 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 bash[64603]: Getting image source signatures 2024-03-26T21:30:17.721 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 bash[64603]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T21:30:17.721 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 bash[64603]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T21:30:17.721 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:17 smithi067 bash[64603]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T21:30:17.791 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-26T21:30:15.283887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1024,"num_read_kb":870,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":626,"ondisk_log_size":626,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240276,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720940,"statfs":{"total":767960285184,"available":767714242560,"internally_reserved":0,"allocated":7434240,"data_stored":4349304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002957"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701782+0000","last_change":"2024-03-26T21:30:03.491344+0000","last_active":"2024-03-26T21:30:06.701782+0000","last_peered":"2024-03-26T21:30:06.701782+0000","last_clean":"2024-03-26T21:30:06.701782+0000","last_became_active":"2024-03-26T21:28:40.879971+0000","last_became_peered":"2024-03-26T21:28:40.879971+0000","last_unstale":"2024-03-26T21:30:06.701782+0000","last_undegraded":"2024-03-26T21:30:06.701782+0000","last_fullsized":"2024-03-26T21:30:06.701782+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.372263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985743+0000","last_change":"2024-03-26T21:28:38.866679+0000","last_active":"2024-03-26T21:30:05.985743+0000","last_peered":"2024-03-26T21:30:05.985743+0000","last_clean":"2024-03-26T21:30:05.985743+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:30:05.985743+0000","last_undegraded":"2024-03-26T21:30:05.985743+0000","last_fullsized":"2024-03-26T21:30:05.985743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721658+0000","last_change":"2024-03-26T21:28:34.844384+0000","last_active":"2024-03-26T21:30:06.721658+0000","last_peered":"2024-03-26T21:30:06.721658+0000","last_clean":"2024-03-26T21:30:06.721658+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:30:06.721658+0000","last_undegraded":"2024-03-26T21:30:06.721658+0000","last_fullsized":"2024-03-26T21:30:06.721658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701691+0000","last_change":"2024-03-26T21:29:51.544731+0000","last_active":"2024-03-26T21:30:06.701691+0000","last_peered":"2024-03-26T21:30:06.701691+0000","last_clean":"2024-03-26T21:30:06.701691+0000","last_became_active":"2024-03-26T21:28:36.841677+0000","last_became_peered":"2024-03-26T21:28:36.841677+0000","last_unstale":"2024-03-26T21:30:06.701691+0000","last_undegraded":"2024-03-26T21:30:06.701691+0000","last_fullsized":"2024-03-26T21:30:06.701691+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.477780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851333+0000","last_change":"2024-03-26T21:29:49.749475+0000","last_active":"2024-03-26T21:30:06.851333+0000","last_peered":"2024-03-26T21:30:06.851333+0000","last_clean":"2024-03-26T21:30:06.851333+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:30:06.851333+0000","last_undegraded":"2024-03-26T21:30:06.851333+0000","last_fullsized":"2024-03-26T21:30:06.851333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:49.749386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:49.749386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.347098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701722+0000","last_change":"2024-03-26T21:29:45.645672+0000","last_active":"2024-03-26T21:30:06.701722+0000","last_peered":"2024-03-26T21:30:06.701722+0000","last_clean":"2024-03-26T21:30:06.701722+0000","last_became_active":"2024-03-26T21:28:38.866501+0000","last_became_peered":"2024-03-26T21:28:38.866501+0000","last_unstale":"2024-03-26T21:30:06.701722+0000","last_undegraded":"2024-03-26T21:30:06.701722+0000","last_fullsized":"2024-03-26T21:30:06.701722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.799670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851726+0000","last_change":"2024-03-26T21:29:50.735279+0000","last_active":"2024-03-26T21:30:06.851726+0000","last_peered":"2024-03-26T21:30:06.851726+0000","last_clean":"2024-03-26T21:30:06.851726+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:30:06.851726+0000","last_undegraded":"2024-03-26T21:30:06.851726+0000","last_fullsized":"2024-03-26T21:30:06.851726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.735199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.735199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.656417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851200+0000","last_change":"2024-03-26T21:30:05.728263+0000","last_active":"2024-03-26T21:30:06.851200+0000","last_peered":"2024-03-26T21:30:06.851200+0000","last_clean":"2024-03-26T21:30:06.851200+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:30:06.851200+0000","last_undegraded":"2024-03-26T21:30:06.851200+0000","last_fullsized":"2024-03-26T21:30:06.851200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:05.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.728097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.255731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721074+0000","last_change":"2024-03-26T21:29:58.804717+0000","last_active":"2024-03-26T21:30:06.721074+0000","last_peered":"2024-03-26T21:30:06.721074+0000","last_clean":"2024-03-26T21:30:06.721074+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:30:06.721074+0000","last_undegraded":"2024-03-26T21:30:06.721074+0000","last_fullsized":"2024-03-26T21:30:06.721074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.804636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.804636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.335495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.494998+0000","last_change":"2024-03-26T21:28:38.872667+0000","last_active":"2024-03-26T21:30:06.494998+0000","last_peered":"2024-03-26T21:30:06.494998+0000","last_clean":"2024-03-26T21:30:06.494998+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:30:06.494998+0000","last_undegraded":"2024-03-26T21:30:06.494998+0000","last_fullsized":"2024-03-26T21:30:06.494998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495465+0000","last_change":"2024-03-26T21:28:34.842302+0000","last_active":"2024-03-26T21:30:06.495465+0000","last_peered":"2024-03-26T21:30:06.495465+0000","last_clean":"2024-03-26T21:30:06.495465+0000","last_became_active":"2024-03-26T21:28:34.841544+0000","last_became_peered":"2024-03-26T21:28:34.841544+0000","last_unstale":"2024-03-26T21:30:06.495465+0000","last_undegraded":"2024-03-26T21:30:06.495465+0000","last_fullsized":"2024-03-26T21:30:06.495465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702126+0000","last_change":"2024-03-26T21:29:52.496860+0000","last_active":"2024-03-26T21:30:06.702126+0000","last_peered":"2024-03-26T21:30:06.702126+0000","last_clean":"2024-03-26T21:30:06.702126+0000","last_became_active":"2024-03-26T21:28:36.847804+0000","last_became_peered":"2024-03-26T21:28:36.847804+0000","last_unstale":"2024-03-26T21:30:06.702126+0000","last_undegraded":"2024-03-26T21:30:06.702126+0000","last_fullsized":"2024-03-26T21:30:06.702126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.032728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.494860+0000","last_change":"2024-03-26T21:28:40.887065+0000","last_active":"2024-03-26T21:30:06.494860+0000","last_peered":"2024-03-26T21:30:06.494860+0000","last_clean":"2024-03-26T21:30:06.494860+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:30:06.494860+0000","last_undegraded":"2024-03-26T21:30:06.494860+0000","last_fullsized":"2024-03-26T21:30:06.494860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:07.539727+0000","last_change":"2024-03-26T21:30:07.539773+0000","last_active":"2024-03-26T21:30:07.539727+0000","last_peered":"2024-03-26T21:30:07.539727+0000","last_clean":"2024-03-26T21:30:07.539727+0000","last_became_active":"2024-03-26T21:28:38.864406+0000","last_became_peered":"2024-03-26T21:28:38.864406+0000","last_unstale":"2024-03-26T21:30:07.539727+0000","last_undegraded":"2024-03-26T21:30:07.539727+0000","last_fullsized":"2024-03-26T21:30:07.539727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:07.539633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.539633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.280041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495346+0000","last_change":"2024-03-26T21:28:34.847255+0000","last_active":"2024-03-26T21:30:06.495346+0000","last_peered":"2024-03-26T21:30:06.495346+0000","last_clean":"2024-03-26T21:30:06.495346+0000","last_became_active":"2024-03-26T21:28:34.846956+0000","last_became_peered":"2024-03-26T21:28:34.846956+0000","last_unstale":"2024-03-26T21:30:06.495346+0000","last_undegraded":"2024-03-26T21:30:06.495346+0000","last_fullsized":"2024-03-26T21:30:06.495346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702296+0000","last_change":"2024-03-26T21:28:36.846687+0000","last_active":"2024-03-26T21:30:06.702296+0000","last_peered":"2024-03-26T21:30:06.702296+0000","last_clean":"2024-03-26T21:30:06.702296+0000","last_became_active":"2024-03-26T21:28:36.846364+0000","last_became_peered":"2024-03-26T21:28:36.846364+0000","last_unstale":"2024-03-26T21:30:06.702296+0000","last_undegraded":"2024-03-26T21:30:06.702296+0000","last_fullsized":"2024-03-26T21:30:06.702296+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702560+0000","last_change":"2024-03-26T21:28:41.114040+0000","last_active":"2024-03-26T21:30:06.702560+0000","last_peered":"2024-03-26T21:30:06.702560+0000","last_clean":"2024-03-26T21:30:06.702560+0000","last_became_active":"2024-03-26T21:28:41.113734+0000","last_became_peered":"2024-03-26T21:28:41.113734+0000","last_unstale":"2024-03-26T21:30:06.702560+0000","last_undegraded":"2024-03-26T21:30:06.702560+0000","last_fullsized":"2024-03-26T21:30:06.702560+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986399+0000","last_change":"2024-03-26T21:29:58.051642+0000","last_active":"2024-03-26T21:30:05.986399+0000","last_peered":"2024-03-26T21:30:05.986399+0000","last_clean":"2024-03-26T21:30:05.986399+0000","last_became_active":"2024-03-26T21:28:38.870984+0000","last_became_peered":"2024-03-26T21:28:38.870984+0000","last_unstale":"2024-03-26T21:30:05.986399+0000","last_undegraded":"2024-03-26T21:30:05.986399+0000","last_fullsized":"2024-03-26T21:30:05.986399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.051556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.051556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.391937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721549+0000","last_change":"2024-03-26T21:30:05.708870+0000","last_active":"2024-03-26T21:30:06.721549+0000","last_peered":"2024-03-26T21:30:06.721549+0000","last_clean":"2024-03-26T21:30:06.721549+0000","last_became_active":"2024-03-26T21:28:34.846913+0000","last_became_peered":"2024-03-26T21:28:34.846913+0000","last_unstale":"2024-03-26T21:30:06.721549+0000","last_undegraded":"2024-03-26T21:30:06.721549+0000","last_fullsized":"2024-03-26T21:30:06.721549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.708767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.708767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.984555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980623+0000","last_change":"2024-03-26T21:29:45.654379+0000","last_active":"2024-03-26T21:30:05.980623+0000","last_peered":"2024-03-26T21:30:05.980623+0000","last_clean":"2024-03-26T21:30:05.980623+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:30:05.980623+0000","last_undegraded":"2024-03-26T21:30:05.980623+0000","last_fullsized":"2024-03-26T21:30:05.980623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:29:45.654238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.654238+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.970402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851656+0000","last_change":"2024-03-26T21:28:41.113863+0000","last_active":"2024-03-26T21:30:06.851656+0000","last_peered":"2024-03-26T21:30:06.851656+0000","last_clean":"2024-03-26T21:30:06.851656+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:30:06.851656+0000","last_undegraded":"2024-03-26T21:30:06.851656+0000","last_fullsized":"2024-03-26T21:30:06.851656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702015+0000","last_change":"2024-03-26T21:29:44.616004+0000","last_active":"2024-03-26T21:30:06.702015+0000","last_peered":"2024-03-26T21:30:06.702015+0000","last_clean":"2024-03-26T21:30:06.702015+0000","last_became_active":"2024-03-26T21:28:38.867256+0000","last_became_peered":"2024-03-26T21:28:38.867256+0000","last_unstale":"2024-03-26T21:30:06.702015+0000","last_undegraded":"2024-03-26T21:30:06.702015+0000","last_fullsized":"2024-03-26T21:30:06.702015+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.353334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702063+0000","last_change":"2024-03-26T21:30:05.464925+0000","last_active":"2024-03-26T21:30:06.702063+0000","last_peered":"2024-03-26T21:30:06.702063+0000","last_clean":"2024-03-26T21:30:06.702063+0000","last_became_active":"2024-03-26T21:28:34.846583+0000","last_became_peered":"2024-03-26T21:28:34.846583+0000","last_unstale":"2024-03-26T21:30:06.702063+0000","last_undegraded":"2024-03-26T21:30:06.702063+0000","last_fullsized":"2024-03-26T21:30:06.702063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.880478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851363+0000","last_change":"2024-03-26T21:28:36.850322+0000","last_active":"2024-03-26T21:30:06.851363+0000","last_peered":"2024-03-26T21:30:06.851363+0000","last_clean":"2024-03-26T21:30:06.851363+0000","last_became_active":"2024-03-26T21:28:36.849968+0000","last_became_peered":"2024-03-26T21:28:36.849968+0000","last_unstale":"2024-03-26T21:30:06.851363+0000","last_undegraded":"2024-03-26T21:30:06.851363+0000","last_fullsized":"2024-03-26T21:30:06.851363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981295+0000","last_change":"2024-03-26T21:29:54.614182+0000","last_active":"2024-03-26T21:30:05.981295+0000","last_peered":"2024-03-26T21:30:05.981295+0000","last_clean":"2024-03-26T21:30:05.981295+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:30:05.981295+0000","last_undegraded":"2024-03-26T21:30:05.981295+0000","last_fullsized":"2024-03-26T21:30:05.981295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:29:54.614064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:54.614064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:06.934031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:10.722267+0000","last_change":"2024-03-26T21:30:10.722294+0000","last_active":"2024-03-26T21:30:10.722267+0000","last_peered":"2024-03-26T21:30:10.722267+0000","last_clean":"2024-03-26T21:30:10.722267+0000","last_became_active":"2024-03-26T21:28:38.867245+0000","last_became_peered":"2024-03-26T21:28:38.867245+0000","last_unstale":"2024-03-26T21:30:10.722267+0000","last_undegraded":"2024-03-26T21:30:10.722267+0000","last_fullsized":"2024-03-26T21:30:10.722267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:10.722203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.722203+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:29.720986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985805+0000","last_change":"2024-03-26T21:30:01.058809+0000","last_active":"2024-03-26T21:30:05.985805+0000","last_peered":"2024-03-26T21:30:05.985805+0000","last_clean":"2024-03-26T21:30:05.985805+0000","last_became_active":"2024-03-26T21:28:34.846408+0000","last_became_peered":"2024-03-26T21:28:34.846408+0000","last_unstale":"2024-03-26T21:30:05.985805+0000","last_undegraded":"2024-03-26T21:30:05.985805+0000","last_fullsized":"2024-03-26T21:30:05.985805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.058714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.303501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721657+0000","last_change":"2024-03-26T21:28:37.143422+0000","last_active":"2024-03-26T21:30:06.721657+0000","last_peered":"2024-03-26T21:30:06.721657+0000","last_clean":"2024-03-26T21:30:06.721657+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:30:06.721657+0000","last_undegraded":"2024-03-26T21:30:06.721657+0000","last_fullsized":"2024-03-26T21:30:06.721657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986140+0000","last_change":"2024-03-26T21:29:45.092723+0000","last_active":"2024-03-26T21:30:05.986140+0000","last_peered":"2024-03-26T21:30:05.986140+0000","last_clean":"2024-03-26T21:30:05.986140+0000","last_became_active":"2024-03-26T21:28:40.881781+0000","last_became_peered":"2024-03-26T21:28:40.881781+0000","last_unstale":"2024-03-26T21:30:05.986140+0000","last_undegraded":"2024-03-26T21:30:05.986140+0000","last_fullsized":"2024-03-26T21:30:05.986140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.160967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721791+0000","last_change":"2024-03-26T21:29:56.787368+0000","last_active":"2024-03-26T21:30:06.721791+0000","last_peered":"2024-03-26T21:30:06.721791+0000","last_clean":"2024-03-26T21:30:06.721791+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:30:06.721791+0000","last_undegraded":"2024-03-26T21:30:06.721791+0000","last_fullsized":"2024-03-26T21:30:06.721791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.787260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.787260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.075256+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.850910+0000","last_change":"2024-03-26T21:30:01.773999+0000","last_active":"2024-03-26T21:30:06.850910+0000","last_peered":"2024-03-26T21:30:06.850910+0000","last_clean":"2024-03-26T21:30:06.850910+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:30:06.850910+0000","last_undegraded":"2024-03-26T21:30:06.850910+0000","last_fullsized":"2024-03-26T21:30:06.850910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.773917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.773917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.422442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497517+0000","last_change":"2024-03-26T21:29:36.821598+0000","last_active":"2024-03-26T21:30:06.497517+0000","last_peered":"2024-03-26T21:30:06.497517+0000","last_clean":"2024-03-26T21:30:06.497517+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:30:06.497517+0000","last_undegraded":"2024-03-26T21:30:06.497517+0000","last_fullsized":"2024-03-26T21:30:06.497517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:29:36.821436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.821436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.432234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:08.535459+0000","last_change":"2024-03-26T21:30:08.535482+0000","last_active":"2024-03-26T21:30:08.535459+0000","last_peered":"2024-03-26T21:30:08.535459+0000","last_clean":"2024-03-26T21:30:08.535459+0000","last_became_active":"2024-03-26T21:28:40.885572+0000","last_became_peered":"2024-03-26T21:28:40.885572+0000","last_unstale":"2024-03-26T21:30:08.535459+0000","last_undegraded":"2024-03-26T21:30:08.535459+0000","last_fullsized":"2024-03-26T21:30:08.535459+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.535399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.535399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:24.167658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.749132+0000","last_change":"2024-03-26T21:30:06.749160+0000","last_active":"2024-03-26T21:30:06.749132+0000","last_peered":"2024-03-26T21:30:06.749132+0000","last_clean":"2024-03-26T21:30:06.749132+0000","last_became_active":"2024-03-26T21:28:38.870142+0000","last_became_peered":"2024-03-26T21:28:38.870142+0000","last_unstale":"2024-03-26T21:30:06.749132+0000","last_undegraded":"2024-03-26T21:30:06.749132+0000","last_fullsized":"2024-03-26T21:30:06.749132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.749046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.749046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:32.228312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721653+0000","last_change":"2024-03-26T21:29:52.893256+0000","last_active":"2024-03-26T21:30:06.721653+0000","last_peered":"2024-03-26T21:30:06.721653+0000","last_clean":"2024-03-26T21:30:06.721653+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:30:06.721653+0000","last_undegraded":"2024-03-26T21:30:06.721653+0000","last_fullsized":"2024-03-26T21:30:06.721653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.893078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.893078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:01.394353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.850962+0000","last_change":"2024-03-26T21:29:57.736516+0000","last_active":"2024-03-26T21:30:06.850962+0000","last_peered":"2024-03-26T21:30:06.850962+0000","last_clean":"2024-03-26T21:30:06.850962+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:30:06.850962+0000","last_undegraded":"2024-03-26T21:30:06.850962+0000","last_fullsized":"2024-03-26T21:30:06.850962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:29:57.736423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:57.736423+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.030206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851107+0000","last_change":"2024-03-26T21:28:41.117667+0000","last_active":"2024-03-26T21:30:06.851107+0000","last_peered":"2024-03-26T21:30:06.851107+0000","last_clean":"2024-03-26T21:30:06.851107+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:30:06.851107+0000","last_undegraded":"2024-03-26T21:30:06.851107+0000","last_fullsized":"2024-03-26T21:30:06.851107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981480+0000","last_change":"2024-03-26T21:28:38.864935+0000","last_active":"2024-03-26T21:30:05.981480+0000","last_peered":"2024-03-26T21:30:05.981480+0000","last_clean":"2024-03-26T21:30:05.981480+0000","last_became_active":"2024-03-26T21:28:38.864421+0000","last_became_peered":"2024-03-26T21:28:38.864421+0000","last_unstale":"2024-03-26T21:30:05.981480+0000","last_undegraded":"2024-03-26T21:30:05.981480+0000","last_fullsized":"2024-03-26T21:30:05.981480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495682+0000","last_change":"2024-03-26T21:30:01.424034+0000","last_active":"2024-03-26T21:30:06.495682+0000","last_peered":"2024-03-26T21:30:06.495682+0000","last_clean":"2024-03-26T21:30:06.495682+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:30:06.495682+0000","last_undegraded":"2024-03-26T21:30:06.495682+0000","last_fullsized":"2024-03-26T21:30:06.495682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.423920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.423920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.782127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851140+0000","last_change":"2024-03-26T21:28:37.142935+0000","last_active":"2024-03-26T21:30:06.851140+0000","last_peered":"2024-03-26T21:30:06.851140+0000","last_clean":"2024-03-26T21:30:06.851140+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:30:06.851140+0000","last_undegraded":"2024-03-26T21:30:06.851140+0000","last_fullsized":"2024-03-26T21:30:06.851140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.720860+0000","last_change":"2024-03-26T21:28:40.889084+0000","last_active":"2024-03-26T21:30:06.720860+0000","last_peered":"2024-03-26T21:30:06.720860+0000","last_clean":"2024-03-26T21:30:06.720860+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:30:06.720860+0000","last_undegraded":"2024-03-26T21:30:06.720860+0000","last_fullsized":"2024-03-26T21:30:06.720860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851366+0000","last_change":"2024-03-26T21:30:04.703815+0000","last_active":"2024-03-26T21:30:06.851366+0000","last_peered":"2024-03-26T21:30:06.851366+0000","last_clean":"2024-03-26T21:30:06.851366+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:30:06.851366+0000","last_undegraded":"2024-03-26T21:30:06.851366+0000","last_fullsized":"2024-03-26T21:30:06.851366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.703726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.703726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.302776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981373+0000","last_change":"2024-03-26T21:29:55.577602+0000","last_active":"2024-03-26T21:30:05.981373+0000","last_peered":"2024-03-26T21:30:05.981373+0000","last_clean":"2024-03-26T21:30:05.981373+0000","last_became_active":"2024-03-26T21:28:34.848518+0000","last_became_peered":"2024-03-26T21:28:34.848518+0000","last_unstale":"2024-03-26T21:30:05.981373+0000","last_undegraded":"2024-03-26T21:30:05.981373+0000","last_fullsized":"2024-03-26T21:30:05.981373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.145785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:10.096181+0000","last_change":"2024-03-26T21:30:10.096212+0000","last_active":"2024-03-26T21:30:10.096181+0000","last_peered":"2024-03-26T21:30:10.096181+0000","last_clean":"2024-03-26T21:30:10.096181+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:30:10.096181+0000","last_undegraded":"2024-03-26T21:30:10.096181+0000","last_fullsized":"2024-03-26T21:30:10.096181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.096107+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.062377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.850944+0000","last_change":"2024-03-26T21:28:40.891718+0000","last_active":"2024-03-26T21:30:06.850944+0000","last_peered":"2024-03-26T21:30:06.850944+0000","last_clean":"2024-03-26T21:30:06.850944+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:30:06.850944+0000","last_undegraded":"2024-03-26T21:30:06.850944+0000","last_fullsized":"2024-03-26T21:30:06.850944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986078+0000","last_change":"2024-03-26T21:30:04.095313+0000","last_active":"2024-03-26T21:30:05.986078+0000","last_peered":"2024-03-26T21:30:05.986078+0000","last_clean":"2024-03-26T21:30:05.986078+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:30:05.986078+0000","last_undegraded":"2024-03-26T21:30:05.986078+0000","last_fullsized":"2024-03-26T21:30:05.986078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.095208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.095208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.418775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702335+0000","last_change":"2024-03-26T21:28:34.844448+0000","last_active":"2024-03-26T21:30:06.702335+0000","last_peered":"2024-03-26T21:30:06.702335+0000","last_clean":"2024-03-26T21:30:06.702335+0000","last_became_active":"2024-03-26T21:28:34.842160+0000","last_became_peered":"2024-03-26T21:28:34.842160+0000","last_unstale":"2024-03-26T21:30:06.702335+0000","last_undegraded":"2024-03-26T21:30:06.702335+0000","last_fullsized":"2024-03-26T21:30:06.702335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702169+0000","last_change":"2024-03-26T21:28:36.846742+0000","last_active":"2024-03-26T21:30:06.702169+0000","last_peered":"2024-03-26T21:30:06.702169+0000","last_clean":"2024-03-26T21:30:06.702169+0000","last_became_active":"2024-03-26T21:28:36.846289+0000","last_became_peered":"2024-03-26T21:28:36.846289+0000","last_unstale":"2024-03-26T21:30:06.702169+0000","last_undegraded":"2024-03-26T21:30:06.702169+0000","last_fullsized":"2024-03-26T21:30:06.702169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721398+0000","last_change":"2024-03-26T21:30:04.667815+0000","last_active":"2024-03-26T21:30:06.721398+0000","last_peered":"2024-03-26T21:30:06.721398+0000","last_clean":"2024-03-26T21:30:06.721398+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:30:06.721398+0000","last_undegraded":"2024-03-26T21:30:06.721398+0000","last_fullsized":"2024-03-26T21:30:06.721398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:30:04.667736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.667736+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.712159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495627+0000","last_change":"2024-03-26T21:29:47.419090+0000","last_active":"2024-03-26T21:30:06.495627+0000","last_peered":"2024-03-26T21:30:06.495627+0000","last_clean":"2024-03-26T21:30:06.495627+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:30:06.495627+0000","last_undegraded":"2024-03-26T21:30:06.495627+0000","last_fullsized":"2024-03-26T21:30:06.495627+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:29:47.418949+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.219762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981168+0000","last_change":"2024-03-26T21:29:39.550921+0000","last_active":"2024-03-26T21:30:05.981168+0000","last_peered":"2024-03-26T21:30:05.981168+0000","last_clean":"2024-03-26T21:30:05.981168+0000","last_became_active":"2024-03-26T21:28:34.839491+0000","last_became_peered":"2024-03-26T21:28:34.839491+0000","last_unstale":"2024-03-26T21:30:05.981168+0000","last_undegraded":"2024-03-26T21:30:05.981168+0000","last_fullsized":"2024-03-26T21:30:05.981168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:53.089443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851430+0000","last_change":"2024-03-26T21:29:55.733996+0000","last_active":"2024-03-26T21:30:06.851430+0000","last_peered":"2024-03-26T21:30:06.851430+0000","last_clean":"2024-03-26T21:30:06.851430+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:30:06.851430+0000","last_undegraded":"2024-03-26T21:30:06.851430+0000","last_fullsized":"2024-03-26T21:30:06.851430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:29:55.733905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.733905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:09.846036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702791+0000","last_change":"2024-03-26T21:28:40.881531+0000","last_active":"2024-03-26T21:30:06.702791+0000","last_peered":"2024-03-26T21:30:06.702791+0000","last_clean":"2024-03-26T21:30:06.702791+0000","last_became_active":"2024-03-26T21:28:40.881234+0000","last_became_peered":"2024-03-26T21:28:40.881234+0000","last_unstale":"2024-03-26T21:30:06.702791+0000","last_undegraded":"2024-03-26T21:30:06.702791+0000","last_fullsized":"2024-03-26T21:30:06.702791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851626+0000","last_change":"2024-03-26T21:28:38.864606+0000","last_active":"2024-03-26T21:30:06.851626+0000","last_peered":"2024-03-26T21:30:06.851626+0000","last_clean":"2024-03-26T21:30:06.851626+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:30:06.851626+0000","last_undegraded":"2024-03-26T21:30:06.851626+0000","last_fullsized":"2024-03-26T21:30:06.851626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721168+0000","last_change":"2024-03-26T21:28:34.841082+0000","last_active":"2024-03-26T21:30:06.721168+0000","last_peered":"2024-03-26T21:30:06.721168+0000","last_clean":"2024-03-26T21:30:06.721168+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:30:06.721168+0000","last_undegraded":"2024-03-26T21:30:06.721168+0000","last_fullsized":"2024-03-26T21:30:06.721168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495825+0000","last_change":"2024-03-26T21:30:02.461420+0000","last_active":"2024-03-26T21:30:06.495825+0000","last_peered":"2024-03-26T21:30:06.495825+0000","last_clean":"2024-03-26T21:30:06.495825+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:30:06.495825+0000","last_undegraded":"2024-03-26T21:30:06.495825+0000","last_fullsized":"2024-03-26T21:30:06.495825+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:30:02.461332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.461332+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.159192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980841+0000","last_change":"2024-03-26T21:30:05.958764+0000","last_active":"2024-03-26T21:30:05.980841+0000","last_peered":"2024-03-26T21:30:05.980841+0000","last_clean":"2024-03-26T21:30:05.980841+0000","last_became_active":"2024-03-26T21:28:40.884963+0000","last_became_peered":"2024-03-26T21:28:40.884963+0000","last_unstale":"2024-03-26T21:30:05.980841+0000","last_undegraded":"2024-03-26T21:30:05.980841+0000","last_fullsized":"2024-03-26T21:30:05.980841+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.958658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.958658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.251328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851908+0000","last_change":"2024-03-26T21:28:38.864418+0000","last_active":"2024-03-26T21:30:06.851908+0000","last_peered":"2024-03-26T21:30:06.851908+0000","last_clean":"2024-03-26T21:30:06.851908+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:30:06.851908+0000","last_undegraded":"2024-03-26T21:30:06.851908+0000","last_fullsized":"2024-03-26T21:30:06.851908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980943+0000","last_change":"2024-03-26T21:28:34.843600+0000","last_active":"2024-03-26T21:30:05.980943+0000","last_peered":"2024-03-26T21:30:05.980943+0000","last_clean":"2024-03-26T21:30:05.980943+0000","last_became_active":"2024-03-26T21:28:34.843175+0000","last_became_peered":"2024-03-26T21:28:34.843175+0000","last_unstale":"2024-03-26T21:30:05.980943+0000","last_undegraded":"2024-03-26T21:30:05.980943+0000","last_fullsized":"2024-03-26T21:30:05.980943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721685+0000","last_change":"2024-03-26T21:30:00.719906+0000","last_active":"2024-03-26T21:30:06.721685+0000","last_peered":"2024-03-26T21:30:06.721685+0000","last_clean":"2024-03-26T21:30:06.721685+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:30:06.721685+0000","last_undegraded":"2024-03-26T21:30:06.721685+0000","last_fullsized":"2024-03-26T21:30:06.721685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:30:00.719791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:00.719791+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.215539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981300+0000","last_change":"2024-03-26T21:28:41.115111+0000","last_active":"2024-03-26T21:30:05.981300+0000","last_peered":"2024-03-26T21:30:05.981300+0000","last_clean":"2024-03-26T21:30:05.981300+0000","last_became_active":"2024-03-26T21:28:41.114831+0000","last_became_peered":"2024-03-26T21:28:41.114831+0000","last_unstale":"2024-03-26T21:30:05.981300+0000","last_undegraded":"2024-03-26T21:30:05.981300+0000","last_fullsized":"2024-03-26T21:30:05.981300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495832+0000","last_change":"2024-03-26T21:30:03.444638+0000","last_active":"2024-03-26T21:30:06.495832+0000","last_peered":"2024-03-26T21:30:06.495832+0000","last_clean":"2024-03-26T21:30:06.495832+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:30:06.495832+0000","last_undegraded":"2024-03-26T21:30:06.495832+0000","last_fullsized":"2024-03-26T21:30:06.495832+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.444530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.444530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:11.858717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.496525+0000","last_change":"2024-03-26T21:29:53.937502+0000","last_active":"2024-03-26T21:30:06.496525+0000","last_peered":"2024-03-26T21:30:06.496525+0000","last_clean":"2024-03-26T21:30:06.496525+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:30:06.496525+0000","last_undegraded":"2024-03-26T21:30:06.496525+0000","last_fullsized":"2024-03-26T21:30:06.496525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:53.937386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:53.937386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.079593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:09.668066+0000","last_change":"2024-03-26T21:30:09.668099+0000","last_active":"2024-03-26T21:30:09.668066+0000","last_peered":"2024-03-26T21:30:09.668066+0000","last_clean":"2024-03-26T21:30:09.668066+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:30:09.668066+0000","last_undegraded":"2024-03-26T21:30:09.668066+0000","last_fullsized":"2024-03-26T21:30:09.668066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:30:09.668002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.668002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:30.607945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495459+0000","last_change":"2024-03-26T21:28:41.114110+0000","last_active":"2024-03-26T21:30:06.495459+0000","last_peered":"2024-03-26T21:30:06.495459+0000","last_clean":"2024-03-26T21:30:06.495459+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:30:06.495459+0000","last_undegraded":"2024-03-26T21:30:06.495459+0000","last_fullsized":"2024-03-26T21:30:06.495459+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981555+0000","last_change":"2024-03-26T21:28:34.843004+0000","last_active":"2024-03-26T21:30:05.981555+0000","last_peered":"2024-03-26T21:30:05.981555+0000","last_clean":"2024-03-26T21:30:05.981555+0000","last_became_active":"2024-03-26T21:28:34.842712+0000","last_became_peered":"2024-03-26T21:28:34.842712+0000","last_unstale":"2024-03-26T21:30:05.981555+0000","last_undegraded":"2024-03-26T21:30:05.981555+0000","last_fullsized":"2024-03-26T21:30:05.981555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:07.493339+0000","last_change":"2024-03-26T21:30:07.493378+0000","last_active":"2024-03-26T21:30:07.493339+0000","last_peered":"2024-03-26T21:30:07.493339+0000","last_clean":"2024-03-26T21:30:07.493339+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:30:07.493339+0000","last_undegraded":"2024-03-26T21:30:07.493339+0000","last_fullsized":"2024-03-26T21:30:07.493339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:07.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.493257+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:31.720965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981178+0000","last_change":"2024-03-26T21:28:38.874954+0000","last_active":"2024-03-26T21:30:05.981178+0000","last_peered":"2024-03-26T21:30:05.981178+0000","last_clean":"2024-03-26T21:30:05.981178+0000","last_became_active":"2024-03-26T21:28:38.874646+0000","last_became_peered":"2024-03-26T21:28:38.874646+0000","last_unstale":"2024-03-26T21:30:05.981178+0000","last_undegraded":"2024-03-26T21:30:05.981178+0000","last_fullsized":"2024-03-26T21:30:05.981178+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851249+0000","last_change":"2024-03-26T21:28:40.888897+0000","last_active":"2024-03-26T21:30:06.851249+0000","last_peered":"2024-03-26T21:30:06.851249+0000","last_clean":"2024-03-26T21:30:06.851249+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:30:06.851249+0000","last_undegraded":"2024-03-26T21:30:06.851249+0000","last_fullsized":"2024-03-26T21:30:06.851249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:08.336674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981525+0000","last_change":"2024-03-26T21:29:44.612815+0000","last_active":"2024-03-26T21:30:05.981525+0000","last_peered":"2024-03-26T21:30:05.981525+0000","last_clean":"2024-03-26T21:30:05.981525+0000","last_became_active":"2024-03-26T21:28:34.848110+0000","last_became_peered":"2024-03-26T21:28:34.848110+0000","last_unstale":"2024-03-26T21:30:05.981525+0000","last_undegraded":"2024-03-26T21:30:05.981525+0000","last_fullsized":"2024-03-26T21:30:05.981525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:52.133060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702295+0000","last_change":"2024-03-26T21:29:42.608574+0000","last_active":"2024-03-26T21:30:06.702295+0000","last_peered":"2024-03-26T21:30:06.702295+0000","last_clean":"2024-03-26T21:30:06.702295+0000","last_became_active":"2024-03-26T21:28:36.847905+0000","last_became_peered":"2024-03-26T21:28:36.847905+0000","last_unstale":"2024-03-26T21:30:06.702295+0000","last_undegraded":"2024-03-26T21:30:06.702295+0000","last_fullsized":"2024-03-26T21:30:06.702295+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.811581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980401+0000","last_change":"2024-03-26T21:30:02.905489+0000","last_active":"2024-03-26T21:30:05.980401+0000","last_peered":"2024-03-26T21:30:05.980401+0000","last_clean":"2024-03-26T21:30:05.980401+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:30:05.980401+0000","last_undegraded":"2024-03-26T21:30:05.980401+0000","last_fullsized":"2024-03-26T21:30:05.980401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:02.905408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.905408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.664056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980719+0000","last_change":"2024-03-26T21:28:40.887205+0000","last_active":"2024-03-26T21:30:05.980719+0000","last_peered":"2024-03-26T21:30:05.980719+0000","last_clean":"2024-03-26T21:30:05.980719+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:30:05.980719+0000","last_undegraded":"2024-03-26T21:30:05.980719+0000","last_fullsized":"2024-03-26T21:30:05.980719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980632+0000","last_change":"2024-03-26T21:30:03.938919+0000","last_active":"2024-03-26T21:30:05.980632+0000","last_peered":"2024-03-26T21:30:05.980632+0000","last_clean":"2024-03-26T21:30:05.980632+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:30:05.980632+0000","last_undegraded":"2024-03-26T21:30:05.980632+0000","last_fullsized":"2024-03-26T21:30:05.980632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.938836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.938836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.038793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981503+0000","last_change":"2024-03-26T21:29:48.625718+0000","last_active":"2024-03-26T21:30:05.981503+0000","last_peered":"2024-03-26T21:30:05.981503+0000","last_clean":"2024-03-26T21:30:05.981503+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:30:05.981503+0000","last_undegraded":"2024-03-26T21:30:05.981503+0000","last_fullsized":"2024-03-26T21:30:05.981503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.625595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.625595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:00.303805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497154+0000","last_change":"2024-03-26T21:29:58.792883+0000","last_active":"2024-03-26T21:30:06.497154+0000","last_peered":"2024-03-26T21:30:06.497154+0000","last_clean":"2024-03-26T21:30:06.497154+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:30:06.497154+0000","last_undegraded":"2024-03-26T21:30:06.497154+0000","last_fullsized":"2024-03-26T21:30:06.497154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:58.792783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.792783+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:02.267914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:08.751772+0000","last_change":"2024-03-26T21:30:08.751794+0000","last_active":"2024-03-26T21:30:08.751772+0000","last_peered":"2024-03-26T21:30:08.751772+0000","last_clean":"2024-03-26T21:30:08.751772+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:30:08.751772+0000","last_undegraded":"2024-03-26T21:30:08.751772+0000","last_fullsized":"2024-03-26T21:30:08.751772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.751700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:08.751700+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.751700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.625207+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980549+0000","last_change":"2024-03-26T21:29:48.738742+0000","last_active":"2024-03-26T21:30:05.980549+0000","last_peered":"2024-03-26T21:30:05.980549+0000","last_clean":"2024-03-26T21:30:05.980549+0000","last_became_active":"2024-03-26T21:28:38.866104+0000","last_became_peered":"2024-03-26T21:28:38.866104+0000","last_unstale":"2024-03-26T21:30:05.980549+0000","last_undegraded":"2024-03-26T21:30:05.980549+0000","last_fullsized":"2024-03-26T21:30:05.980549+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.530253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495545+0000","last_change":"2024-03-26T21:28:34.841686+0000","last_active":"2024-03-26T21:30:06.495545+0000","last_peered":"2024-03-26T21:30:06.495545+0000","last_clean":"2024-03-26T21:30:06.495545+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:30:06.495545+0000","last_undegraded":"2024-03-26T21:30:06.495545+0000","last_fullsized":"2024-03-26T21:30:06.495545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721732+0000","last_change":"2024-03-26T21:28:36.845693+0000","last_active":"2024-03-26T21:30:06.721732+0000","last_peered":"2024-03-26T21:30:06.721732+0000","last_clean":"2024-03-26T21:30:06.721732+0000","last_became_active":"2024-03-26T21:28:36.845358+0000","last_became_peered":"2024-03-26T21:28:36.845358+0000","last_unstale":"2024-03-26T21:30:06.721732+0000","last_undegraded":"2024-03-26T21:30:06.721732+0000","last_fullsized":"2024-03-26T21:30:06.721732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851585+0000","last_change":"2024-03-26T21:29:42.726419+0000","last_active":"2024-03-26T21:30:06.851585+0000","last_peered":"2024-03-26T21:30:06.851585+0000","last_clean":"2024-03-26T21:30:06.851585+0000","last_became_active":"2024-03-26T21:28:40.884694+0000","last_became_peered":"2024-03-26T21:28:40.884694+0000","last_unstale":"2024-03-26T21:30:06.851585+0000","last_undegraded":"2024-03-26T21:30:06.851585+0000","last_fullsized":"2024-03-26T21:30:06.851585+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.138620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.496350+0000","last_change":"2024-03-26T21:30:06.496381+0000","last_active":"2024-03-26T21:30:06.496350+0000","last_peered":"2024-03-26T21:30:06.496350+0000","last_clean":"2024-03-26T21:30:06.496350+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:30:06.496350+0000","last_undegraded":"2024-03-26T21:30:06.496350+0000","last_fullsized":"2024-03-26T21:30:06.496350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:30:06.496257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.496257+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.713708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701920+0000","last_change":"2024-03-26T21:28:34.842465+0000","last_active":"2024-03-26T21:30:06.701920+0000","last_peered":"2024-03-26T21:30:06.701920+0000","last_clean":"2024-03-26T21:30:06.701920+0000","last_became_active":"2024-03-26T21:28:34.842130+0000","last_became_peered":"2024-03-26T21:28:34.842130+0000","last_unstale":"2024-03-26T21:30:06.701920+0000","last_undegraded":"2024-03-26T21:30:06.701920+0000","last_fullsized":"2024-03-26T21:30:06.701920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495166+0000","last_change":"2024-03-26T21:29:56.452291+0000","last_active":"2024-03-26T21:30:06.495166+0000","last_peered":"2024-03-26T21:30:06.495166+0000","last_clean":"2024-03-26T21:30:06.495166+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:30:06.495166+0000","last_undegraded":"2024-03-26T21:30:06.495166+0000","last_fullsized":"2024-03-26T21:30:06.495166+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:29:56.452170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.452170+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:14.545743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721937+0000","last_change":"2024-03-26T21:30:06.721957+0000","last_active":"2024-03-26T21:30:06.721937+0000","last_peered":"2024-03-26T21:30:06.721937+0000","last_clean":"2024-03-26T21:30:06.721937+0000","last_became_active":"2024-03-26T21:28:38.873196+0000","last_became_peered":"2024-03-26T21:28:38.873196+0000","last_unstale":"2024-03-26T21:30:06.721937+0000","last_undegraded":"2024-03-26T21:30:06.721937+0000","last_fullsized":"2024-03-26T21:30:06.721937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.721880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.721880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.415483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981604+0000","last_change":"2024-03-26T21:28:41.114073+0000","last_active":"2024-03-26T21:30:05.981604+0000","last_peered":"2024-03-26T21:30:05.981604+0000","last_clean":"2024-03-26T21:30:05.981604+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:30:05.981604+0000","last_undegraded":"2024-03-26T21:30:05.981604+0000","last_fullsized":"2024-03-26T21:30:05.981604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985798+0000","last_change":"2024-03-26T21:29:36.249432+0000","last_active":"2024-03-26T21:30:05.985798+0000","last_peered":"2024-03-26T21:30:05.985798+0000","last_clean":"2024-03-26T21:30:05.985798+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:30:05.985798+0000","last_undegraded":"2024-03-26T21:30:05.985798+0000","last_fullsized":"2024-03-26T21:30:05.985798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:36.249302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.249302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:39.117721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980486+0000","last_change":"2024-03-26T21:29:41.695430+0000","last_active":"2024-03-26T21:30:05.980486+0000","last_peered":"2024-03-26T21:30:05.980486+0000","last_clean":"2024-03-26T21:30:05.980486+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:30:05.980486+0000","last_undegraded":"2024-03-26T21:30:05.980486+0000","last_fullsized":"2024-03-26T21:30:05.980486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.695322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.695322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.842596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986561+0000","last_change":"2024-03-26T21:29:55.054708+0000","last_active":"2024-03-26T21:30:05.986561+0000","last_peered":"2024-03-26T21:30:05.986561+0000","last_clean":"2024-03-26T21:30:05.986561+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:30:05.986561+0000","last_undegraded":"2024-03-26T21:30:05.986561+0000","last_fullsized":"2024-03-26T21:30:05.986561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.054601+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.383564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.957036+0000","last_change":"2024-03-26T21:30:06.957057+0000","last_active":"2024-03-26T21:30:06.957036+0000","last_peered":"2024-03-26T21:30:06.957036+0000","last_clean":"2024-03-26T21:30:06.957036+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:30:06.957036+0000","last_undegraded":"2024-03-26T21:30:06.957036+0000","last_fullsized":"2024-03-26T21:30:06.957036+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.956977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.956977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:08.373708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:08.473059+0000","last_change":"2024-03-26T21:30:08.473082+0000","last_active":"2024-03-26T21:30:08.473059+0000","last_peered":"2024-03-26T21:30:08.473059+0000","last_clean":"2024-03-26T21:30:08.473059+0000","last_became_active":"2024-03-26T21:28:40.886965+0000","last_became_peered":"2024-03-26T21:28:40.886965+0000","last_unstale":"2024-03-26T21:30:08.473059+0000","last_undegraded":"2024-03-26T21:30:08.473059+0000","last_fullsized":"2024-03-26T21:30:08.473059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.472985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.472985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.463284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":310,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980852+0000","last_change":"2024-03-26T21:28:42.275445+0000","last_active":"2024-03-26T21:30:05.980852+0000","last_peered":"2024-03-26T21:30:05.980852+0000","last_clean":"2024-03-26T21:30:05.980852+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:30:05.980852+0000","last_undegraded":"2024-03-26T21:30:05.980852+0000","last_fullsized":"2024-03-26T21:30:05.980852+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T21:28:42.275350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:28:42.275350+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851187+0000","last_change":"2024-03-26T21:29:38.690870+0000","last_active":"2024-03-26T21:30:06.851187+0000","last_peered":"2024-03-26T21:30:06.851187+0000","last_clean":"2024-03-26T21:30:06.851187+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:30:06.851187+0000","last_undegraded":"2024-03-26T21:30:06.851187+0000","last_fullsized":"2024-03-26T21:30:06.851187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.690779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.690779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.230074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"88'24","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497472+0000","last_change":"2024-03-26T21:29:55.903181+0000","last_active":"2024-03-26T21:30:06.497472+0000","last_peered":"2024-03-26T21:30:06.497472+0000","last_clean":"2024-03-26T21:30:06.497472+0000","last_became_active":"2024-03-26T21:28:37.140774+0000","last_became_peered":"2024-03-26T21:28:37.140774+0000","last_unstale":"2024-03-26T21:30:06.497472+0000","last_undegraded":"2024-03-26T21:30:06.497472+0000","last_fullsized":"2024-03-26T21:30:06.497472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:13.673385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981733+0000","last_change":"2024-03-26T21:28:38.868716+0000","last_active":"2024-03-26T21:30:05.981733+0000","last_peered":"2024-03-26T21:30:05.981733+0000","last_clean":"2024-03-26T21:30:05.981733+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:30:05.981733+0000","last_undegraded":"2024-03-26T21:30:05.981733+0000","last_fullsized":"2024-03-26T21:30:05.981733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986306+0000","last_change":"2024-03-26T21:28:40.887424+0000","last_active":"2024-03-26T21:30:05.986306+0000","last_peered":"2024-03-26T21:30:05.986306+0000","last_clean":"2024-03-26T21:30:05.986306+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:30:05.986306+0000","last_undegraded":"2024-03-26T21:30:05.986306+0000","last_fullsized":"2024-03-26T21:30:05.986306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721086+0000","last_change":"2024-03-26T21:28:34.843666+0000","last_active":"2024-03-26T21:30:06.721086+0000","last_peered":"2024-03-26T21:30:06.721086+0000","last_clean":"2024-03-26T21:30:06.721086+0000","last_became_active":"2024-03-26T21:28:34.843392+0000","last_became_peered":"2024-03-26T21:28:34.843392+0000","last_unstale":"2024-03-26T21:30:06.721086+0000","last_undegraded":"2024-03-26T21:30:06.721086+0000","last_fullsized":"2024-03-26T21:30:06.721086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":159,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:09.485367+0000","last_change":"2024-03-26T21:28:38.848730+0000","last_active":"2024-03-26T21:30:09.485367+0000","last_peered":"2024-03-26T21:30:09.485367+0000","last_clean":"2024-03-26T21:30:09.485367+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:30:09.485367+0000","last_undegraded":"2024-03-26T21:30:09.485367+0000","last_fullsized":"2024-03-26T21:30:09.485367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721405+0000","last_change":"2024-03-26T21:29:38.920799+0000","last_active":"2024-03-26T21:30:06.721405+0000","last_peered":"2024-03-26T21:30:06.721405+0000","last_clean":"2024-03-26T21:30:06.721405+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:30:06.721405+0000","last_undegraded":"2024-03-26T21:30:06.721405+0000","last_fullsized":"2024-03-26T21:30:06.721405+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:38.920701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.920701+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:43.589043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495866+0000","last_change":"2024-03-26T21:28:38.870606+0000","last_active":"2024-03-26T21:30:06.495866+0000","last_peered":"2024-03-26T21:30:06.495866+0000","last_clean":"2024-03-26T21:30:06.495866+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:30:06.495866+0000","last_undegraded":"2024-03-26T21:30:06.495866+0000","last_fullsized":"2024-03-26T21:30:06.495866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:07.288276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985698+0000","last_change":"2024-03-26T21:30:03.078966+0000","last_active":"2024-03-26T21:30:05.985698+0000","last_peered":"2024-03-26T21:30:05.985698+0000","last_clean":"2024-03-26T21:30:05.985698+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:30:05.985698+0000","last_undegraded":"2024-03-26T21:30:05.985698+0000","last_fullsized":"2024-03-26T21:30:05.985698+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.078885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.078885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:20.097118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.985977+0000","last_change":"2024-03-26T21:29:42.167843+0000","last_active":"2024-03-26T21:30:05.985977+0000","last_peered":"2024-03-26T21:30:05.985977+0000","last_clean":"2024-03-26T21:30:05.985977+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:30:05.985977+0000","last_undegraded":"2024-03-26T21:30:05.985977+0000","last_fullsized":"2024-03-26T21:30:05.985977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.167719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:45.815629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721598+0000","last_change":"2024-03-26T21:29:51.863137+0000","last_active":"2024-03-26T21:30:06.721598+0000","last_peered":"2024-03-26T21:30:06.721598+0000","last_clean":"2024-03-26T21:30:06.721598+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:30:06.721598+0000","last_undegraded":"2024-03-26T21:30:06.721598+0000","last_fullsized":"2024-03-26T21:30:06.721598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:51.862960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.862960+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:12.041689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497165+0000","last_change":"2024-03-26T21:29:52.956781+0000","last_active":"2024-03-26T21:30:06.497165+0000","last_peered":"2024-03-26T21:30:06.497165+0000","last_clean":"2024-03-26T21:30:06.497165+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:30:06.497165+0000","last_undegraded":"2024-03-26T21:30:06.497165+0000","last_fullsized":"2024-03-26T21:30:06.497165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.956669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.956669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:54.531858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.720861+0000","last_change":"2024-03-26T21:28:40.890739+0000","last_active":"2024-03-26T21:30:06.720861+0000","last_peered":"2024-03-26T21:30:06.720861+0000","last_clean":"2024-03-26T21:30:06.720861+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:30:06.720861+0000","last_undegraded":"2024-03-26T21:30:06.720861+0000","last_fullsized":"2024-03-26T21:30:06.720861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495644+0000","last_change":"2024-03-26T21:29:38.189988+0000","last_active":"2024-03-26T21:30:06.495644+0000","last_peered":"2024-03-26T21:30:06.495644+0000","last_clean":"2024-03-26T21:30:06.495644+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:30:06.495644+0000","last_undegraded":"2024-03-26T21:30:06.495644+0000","last_fullsized":"2024-03-26T21:30:06.495644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:38.189877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.189877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:38.248593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986154+0000","last_change":"2024-03-26T21:30:02.076222+0000","last_active":"2024-03-26T21:30:05.986154+0000","last_peered":"2024-03-26T21:30:05.986154+0000","last_clean":"2024-03-26T21:30:05.986154+0000","last_became_active":"2024-03-26T21:28:36.841759+0000","last_became_peered":"2024-03-26T21:28:36.841759+0000","last_unstale":"2024-03-26T21:30:05.986154+0000","last_undegraded":"2024-03-26T21:30:05.986154+0000","last_fullsized":"2024-03-26T21:30:05.986154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:02.076064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.076064+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:14.477601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721480+0000","last_change":"2024-03-26T21:28:38.869750+0000","last_active":"2024-03-26T21:30:06.721480+0000","last_peered":"2024-03-26T21:30:06.721480+0000","last_clean":"2024-03-26T21:30:06.721480+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:30:06.721480+0000","last_undegraded":"2024-03-26T21:30:06.721480+0000","last_fullsized":"2024-03-26T21:30:06.721480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.981576+0000","last_change":"2024-03-26T21:28:41.115853+0000","last_active":"2024-03-26T21:30:05.981576+0000","last_peered":"2024-03-26T21:30:05.981576+0000","last_clean":"2024-03-26T21:30:05.981576+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:30:05.981576+0000","last_undegraded":"2024-03-26T21:30:05.981576+0000","last_fullsized":"2024-03-26T21:30:05.981576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.721520+0000","last_change":"2024-03-26T21:28:34.846498+0000","last_active":"2024-03-26T21:30:06.721520+0000","last_peered":"2024-03-26T21:30:06.721520+0000","last_clean":"2024-03-26T21:30:06.721520+0000","last_became_active":"2024-03-26T21:28:34.846219+0000","last_became_peered":"2024-03-26T21:28:34.846219+0000","last_unstale":"2024-03-26T21:30:06.721520+0000","last_undegraded":"2024-03-26T21:30:06.721520+0000","last_fullsized":"2024-03-26T21:30:06.721520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:09.414791+0000","last_change":"2024-03-26T21:30:09.414832+0000","last_active":"2024-03-26T21:30:09.414791+0000","last_peered":"2024-03-26T21:30:09.414791+0000","last_clean":"2024-03-26T21:30:09.414791+0000","last_became_active":"2024-03-26T21:28:36.839780+0000","last_became_peered":"2024-03-26T21:28:36.839780+0000","last_unstale":"2024-03-26T21:30:09.414791+0000","last_undegraded":"2024-03-26T21:30:09.414791+0000","last_fullsized":"2024-03-26T21:30:09.414791+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:14.751788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851157+0000","last_change":"2024-03-26T21:29:59.694526+0000","last_active":"2024-03-26T21:30:06.851157+0000","last_peered":"2024-03-26T21:30:06.851157+0000","last_clean":"2024-03-26T21:30:06.851157+0000","last_became_active":"2024-03-26T21:28:38.864364+0000","last_became_peered":"2024-03-26T21:28:38.864364+0000","last_unstale":"2024-03-26T21:30:06.851157+0000","last_undegraded":"2024-03-26T21:30:06.851157+0000","last_fullsized":"2024-03-26T21:30:06.851157+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:29:59.694434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:59.694434+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.427100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701849+0000","last_change":"2024-03-26T21:29:48.632454+0000","last_active":"2024-03-26T21:30:06.701849+0000","last_peered":"2024-03-26T21:30:06.701849+0000","last_clean":"2024-03-26T21:30:06.701849+0000","last_became_active":"2024-03-26T21:28:40.886497+0000","last_became_peered":"2024-03-26T21:28:40.886497+0000","last_unstale":"2024-03-26T21:30:06.701849+0000","last_undegraded":"2024-03-26T21:30:06.701849+0000","last_fullsized":"2024-03-26T21:30:06.701849+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.661591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851757+0000","last_change":"2024-03-26T21:29:56.690968+0000","last_active":"2024-03-26T21:30:06.851757+0000","last_peered":"2024-03-26T21:30:06.851757+0000","last_clean":"2024-03-26T21:30:06.851757+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:30:06.851757+0000","last_undegraded":"2024-03-26T21:30:06.851757+0000","last_fullsized":"2024-03-26T21:30:06.851757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.690884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.690884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.513440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851081+0000","last_change":"2024-03-26T21:28:37.143706+0000","last_active":"2024-03-26T21:30:06.851081+0000","last_peered":"2024-03-26T21:30:06.851081+0000","last_clean":"2024-03-26T21:30:06.851081+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:30:06.851081+0000","last_undegraded":"2024-03-26T21:30:06.851081+0000","last_fullsized":"2024-03-26T21:30:06.851081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701785+0000","last_change":"2024-03-26T21:28:38.863832+0000","last_active":"2024-03-26T21:30:06.701785+0000","last_peered":"2024-03-26T21:30:06.701785+0000","last_clean":"2024-03-26T21:30:06.701785+0000","last_became_active":"2024-03-26T21:28:38.863503+0000","last_became_peered":"2024-03-26T21:28:38.863503+0000","last_unstale":"2024-03-26T21:30:06.701785+0000","last_undegraded":"2024-03-26T21:30:06.701785+0000","last_fullsized":"2024-03-26T21:30:06.701785+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495276+0000","last_change":"2024-03-26T21:30:04.491878+0000","last_active":"2024-03-26T21:30:06.495276+0000","last_peered":"2024-03-26T21:30:06.495276+0000","last_clean":"2024-03-26T21:30:06.495276+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:30:06.495276+0000","last_undegraded":"2024-03-26T21:30:06.495276+0000","last_fullsized":"2024-03-26T21:30:06.495276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.491791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.670339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702740+0000","last_change":"2024-03-26T21:28:38.871742+0000","last_active":"2024-03-26T21:30:06.702740+0000","last_peered":"2024-03-26T21:30:06.702740+0000","last_clean":"2024-03-26T21:30:06.702740+0000","last_became_active":"2024-03-26T21:28:38.871425+0000","last_became_peered":"2024-03-26T21:28:38.871425+0000","last_unstale":"2024-03-26T21:30:06.702740+0000","last_undegraded":"2024-03-26T21:30:06.702740+0000","last_fullsized":"2024-03-26T21:30:06.702740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851005+0000","last_change":"2024-03-26T21:29:41.689438+0000","last_active":"2024-03-26T21:30:06.851005+0000","last_peered":"2024-03-26T21:30:06.851005+0000","last_clean":"2024-03-26T21:30:06.851005+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:30:06.851005+0000","last_undegraded":"2024-03-26T21:30:06.851005+0000","last_fullsized":"2024-03-26T21:30:06.851005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.689332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.689332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.552879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:11.086943+0000","last_change":"2024-03-26T21:30:11.086983+0000","last_active":"2024-03-26T21:30:11.086943+0000","last_peered":"2024-03-26T21:30:11.086943+0000","last_clean":"2024-03-26T21:30:11.086943+0000","last_became_active":"2024-03-26T21:28:36.847705+0000","last_became_peered":"2024-03-26T21:28:36.847705+0000","last_unstale":"2024-03-26T21:30:11.086943+0000","last_undegraded":"2024-03-26T21:30:11.086943+0000","last_fullsized":"2024-03-26T21:30:11.086943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:11.086876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.086876+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.171215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851326+0000","last_change":"2024-03-26T21:28:41.113828+0000","last_active":"2024-03-26T21:30:06.851326+0000","last_peered":"2024-03-26T21:30:06.851326+0000","last_clean":"2024-03-26T21:30:06.851326+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:30:06.851326+0000","last_undegraded":"2024-03-26T21:30:06.851326+0000","last_fullsized":"2024-03-26T21:30:06.851326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.495740+0000","last_change":"2024-03-26T21:28:38.872061+0000","last_active":"2024-03-26T21:30:06.495740+0000","last_peered":"2024-03-26T21:30:06.495740+0000","last_clean":"2024-03-26T21:30:06.495740+0000","last_became_active":"2024-03-26T21:28:38.869931+0000","last_became_peered":"2024-03-26T21:28:38.869931+0000","last_unstale":"2024-03-26T21:30:06.495740+0000","last_undegraded":"2024-03-26T21:30:06.495740+0000","last_fullsized":"2024-03-26T21:30:06.495740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.701792+0000","last_change":"2024-03-26T21:29:50.575715+0000","last_active":"2024-03-26T21:30:06.701792+0000","last_peered":"2024-03-26T21:30:06.701792+0000","last_clean":"2024-03-26T21:30:06.701792+0000","last_became_active":"2024-03-26T21:28:34.842398+0000","last_became_peered":"2024-03-26T21:28:34.842398+0000","last_unstale":"2024-03-26T21:30:06.701792+0000","last_undegraded":"2024-03-26T21:30:06.701792+0000","last_fullsized":"2024-03-26T21:30:06.701792+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:55.357015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702555+0000","last_change":"2024-03-26T21:30:01.544846+0000","last_active":"2024-03-26T21:30:06.702555+0000","last_peered":"2024-03-26T21:30:06.702555+0000","last_clean":"2024-03-26T21:30:06.702555+0000","last_became_active":"2024-03-26T21:28:36.841703+0000","last_became_peered":"2024-03-26T21:28:36.841703+0000","last_unstale":"2024-03-26T21:30:06.702555+0000","last_undegraded":"2024-03-26T21:30:06.702555+0000","last_fullsized":"2024-03-26T21:30:06.702555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:04.152930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980913+0000","last_change":"2024-03-26T21:30:04.923415+0000","last_active":"2024-03-26T21:30:05.980913+0000","last_peered":"2024-03-26T21:30:05.980913+0000","last_clean":"2024-03-26T21:30:05.980913+0000","last_became_active":"2024-03-26T21:28:40.884823+0000","last_became_peered":"2024-03-26T21:28:40.884823+0000","last_unstale":"2024-03-26T21:30:05.980913+0000","last_undegraded":"2024-03-26T21:30:05.980913+0000","last_fullsized":"2024-03-26T21:30:05.980913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.923329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.736283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.986297+0000","last_change":"2024-03-26T21:29:40.185232+0000","last_active":"2024-03-26T21:30:05.986297+0000","last_peered":"2024-03-26T21:30:05.986297+0000","last_clean":"2024-03-26T21:30:05.986297+0000","last_became_active":"2024-03-26T21:28:38.867110+0000","last_became_peered":"2024-03-26T21:28:38.867110+0000","last_unstale":"2024-03-26T21:30:05.986297+0000","last_undegraded":"2024-03-26T21:30:05.986297+0000","last_fullsized":"2024-03-26T21:30:05.986297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:05.295284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.497012+0000","last_change":"2024-03-26T21:29:39.867915+0000","last_active":"2024-03-26T21:30:06.497012+0000","last_peered":"2024-03-26T21:30:06.497012+0000","last_clean":"2024-03-26T21:30:06.497012+0000","last_became_active":"2024-03-26T21:28:34.846385+0000","last_became_peered":"2024-03-26T21:28:34.846385+0000","last_unstale":"2024-03-26T21:30:06.497012+0000","last_undegraded":"2024-03-26T21:30:06.497012+0000","last_fullsized":"2024-03-26T21:30:06.497012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:51.580336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702500+0000","last_change":"2024-03-26T21:28:36.842153+0000","last_active":"2024-03-26T21:30:06.702500+0000","last_peered":"2024-03-26T21:30:06.702500+0000","last_clean":"2024-03-26T21:30:06.702500+0000","last_became_active":"2024-03-26T21:28:36.841769+0000","last_became_peered":"2024-03-26T21:28:36.841769+0000","last_unstale":"2024-03-26T21:30:06.702500+0000","last_undegraded":"2024-03-26T21:30:06.702500+0000","last_fullsized":"2024-03-26T21:30:06.702500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702702+0000","last_change":"2024-03-26T21:29:58.537245+0000","last_active":"2024-03-26T21:30:06.702702+0000","last_peered":"2024-03-26T21:30:06.702702+0000","last_clean":"2024-03-26T21:30:06.702702+0000","last_became_active":"2024-03-26T21:28:40.879974+0000","last_became_peered":"2024-03-26T21:28:40.879974+0000","last_unstale":"2024-03-26T21:30:06.702702+0000","last_undegraded":"2024-03-26T21:30:06.702702+0000","last_fullsized":"2024-03-26T21:30:06.702702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.657997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:05.980720+0000","last_change":"2024-03-26T21:30:01.872594+0000","last_active":"2024-03-26T21:30:05.980720+0000","last_peered":"2024-03-26T21:30:05.980720+0000","last_clean":"2024-03-26T21:30:05.980720+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:30:05.980720+0000","last_undegraded":"2024-03-26T21:30:05.980720+0000","last_fullsized":"2024-03-26T21:30:05.980720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:01.872464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.872464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:10.137769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.851450+0000","last_change":"2024-03-26T21:28:34.848313+0000","last_active":"2024-03-26T21:30:06.851450+0000","last_peered":"2024-03-26T21:30:06.851450+0000","last_clean":"2024-03-26T21:30:06.851450+0000","last_became_active":"2024-03-26T21:28:34.848009+0000","last_became_peered":"2024-03-26T21:28:34.848009+0000","last_unstale":"2024-03-26T21:30:06.851450+0000","last_undegraded":"2024-03-26T21:30:06.851450+0000","last_fullsized":"2024-03-26T21:30:06.851450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T21:30:06.702083+0000","last_change":"2024-03-26T21:29:37.647253+0000","last_active":"2024-03-26T21:30:06.702083+0000","last_peered":"2024-03-26T21:30:06.702083+0000","last_clean":"2024-03-26T21:30:06.702083+0000","last_became_active":"2024-03-26T21:28:36.847902+0000","last_became_peered":"2024-03-26T21:28:36.847902+0000","last_unstale":"2024-03-26T21:30:06.702083+0000","last_undegraded":"2024-03-26T21:30:06.702083+0000","last_fullsized":"2024-03-26T21:30:06.702083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:30:48.419033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070488,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1306624,"data_stored":913849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364830,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28260,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716892,"statfs":{"total":95995035648,"available":95966097408,"internally_reserved":0,"allocated":1273856,"data_stored":911437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":733184,"data_stored":322932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1306624,"data_stored":913155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31836,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":675840,"data_stored":321184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":720896,"data_stored":321973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":716800,"data_stored":323021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":700416,"data_stored":321753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:30:17.797 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T21:30:17.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump 2024-03-26T21:30:18.052 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:18.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[29406]: Deploying daemon node-exporter.b on smithi067 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[29406]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[40587]: Deploying daemon node-exporter.b on smithi067 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:30:18.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-mon[40587]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:18 smithi067 ceph-mon[33346]: Deploying daemon node-exporter.b on smithi067 2024-03-26T21:30:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:18 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:30:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:18 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:30:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:18 smithi067 ceph-mon[33346]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:30:19.195 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:30:19.195 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: 3.1 deep-scrub starts 2024-03-26T21:30:19.195 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: 3.1 deep-scrub ok 2024-03-26T21:30:19.195 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: 3.1 deep-scrub starts 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: 3.1 deep-scrub ok 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:19 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:30:19.196 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:30:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:19.335 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:30:19.335 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: 3.1 deep-scrub starts 2024-03-26T21:30:19.335 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: 3.1 deep-scrub ok 2024-03-26T21:30:19.335 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:30:19.335 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:30:19.336 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:30:19.336 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:30:19.336 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 bash[64603]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T21:30:19.336 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 bash[64603]: Writing manifest to image destination 2024-03-26T21:30:19.924 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 podman[64603]: 2024-03-26 21:30:19.606748002 +0000 UTC m=+2.364958769 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T21:30:19.925 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 podman[64603]: 2024-03-26 21:30:19.705270274 +0000 UTC m=+2.463481041 container create 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:30:19.925 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 podman[64603]: 2024-03-26 21:30:19.932477726 +0000 UTC m=+2.690688494 container init 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 92 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-03-26T21:22:56.673257+0000 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2024-03-26T21:30:19.050225+0000 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:crush_version 18 2024-03-26T21:30:20.026 INFO:teuthology.orchestra.run.smithi055.stdout:full_ratio 0.95 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:backfillfull_ratio 0.9 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:nearfull_ratio 0.85 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:require_min_compat_client luminous 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:min_compat_client luminous 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:require_osd_release squid 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:stretch_mode_enabled false 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:max_osd 8 2024-03-26T21:30:20.027 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6801/317381212 v2:172.21.15.55:6802/317381212 exists,up 4f487071-c151-406b-b25b-3135cb3c63ae 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6805/3733554168 v2:172.21.15.55:6806/3733554168 exists,up 4fbc90fe-926a-4ac6-b5f0-d483ba4c09db 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6809/3059242923 v2:172.21.15.55:6810/3059242923 exists,up e2fe8018-1060-499c-a7e6-56d4da0eba7b 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6813/3851321468 v2:172.21.15.55:6814/3851321468 exists,up 30637a8d-5c1b-4ede-8214-a4595c875d1b 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6800/1777724250 v2:172.21.15.67:6801/1777724250 exists,up cd2d68be-4b10-4242-826b-e085c2ce5598 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 up in weight 1 up_from 41 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6804/278711358 v2:172.21.15.67:6805/278711358 exists,up 7468b94a-3216-4bc2-9b57-1d93e7c0b213 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6808/3926251639 v2:172.21.15.67:6809/3926251639 exists,up d47c79f3-96bf-46b6-a7f3-a315446d04b9 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6812/2981645260 v2:172.21.15.67:6813/2981645260 exists,up f10b98b8-887d-4f83-8bc9-99f5c302dc55 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:pg_upmap_items 4.4 [1,5] 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:pg_upmap_items 4.6 [1,5] 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:pg_upmap_items 4.17 [1,2] 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/2649740776 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/4141219490 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.028 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:6800/147833426 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/2854772543 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:6800/2804789175 expires 2024-03-27T21:23:57.787077+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/2199694469 expires 2024-03-27T21:23:57.787077+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/1614234257 expires 2024-03-27T21:23:30.740110+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/2834138692 expires 2024-03-27T21:23:30.740110+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/1740219825 expires 2024-03-27T21:23:30.740110+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/268664696 expires 2024-03-27T21:23:57.787077+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:6800/536717527 expires 2024-03-27T21:23:30.740110+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/966571165 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/2757111431 expires 2024-03-27T21:29:15.251747+0000 2024-03-26T21:30:20.029 INFO:teuthology.orchestra.run.smithi055.stdout:blocklist 172.21.15.55:0/559658018 expires 2024-03-27T21:23:57.787077+0000 2024-03-26T21:30:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:20 smithi067 ceph-mon[33346]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:30:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:20 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:30:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:20 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:30:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1918336221' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 podman[64603]: 2024-03-26 21:30:19.935549298 +0000 UTC m=+2.693760066 container start 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.939Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.939Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.939Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T21:30:20.240 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T21:30:20.241 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T21:30:20.242 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.940Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T21:30:20.243 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b[64653]: ts=2024-03-26T21:30:19.941Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:20 smithi067 bash[64603]: 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 2024-03-26T21:30:20.244 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:30:20 smithi067 systemd[1]: Started Ceph node-exporter.b for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[40587]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:30:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:30:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:30:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1918336221' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:30:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[29406]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T21:30:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:30:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:30:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1918336221' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T21:30:20.723 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6801/317381212 v2:172.21.15.55:6802/317381212 exists,up 4f487071-c151-406b-b25b-3135cb3c63ae', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6805/3733554168 v2:172.21.15.55:6806/3733554168 exists,up 4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6809/3059242923 v2:172.21.15.55:6810/3059242923 exists,up e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.55:6813/3851321468 v2:172.21.15.55:6814/3851321468 exists,up 30637a8d-5c1b-4ede-8214-a4595c875d1b', 'osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6800/1777724250 v2:172.21.15.67:6801/1777724250 exists,up cd2d68be-4b10-4242-826b-e085c2ce5598', 'osd.5 up in weight 1 up_from 41 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6804/278711358 v2:172.21.15.67:6805/278711358 exists,up 7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6808/3926251639 v2:172.21.15.67:6809/3926251639 exists,up d47c79f3-96bf-46b6-a7f3-a315446d04b9', 'osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.67:6812/2981645260 v2:172.21.15.67:6813/2981645260 exists,up f10b98b8-887d-4f83-8bc9-99f5c302dc55'] 2024-03-26T21:30:20.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon mon.a config show 2024-03-26T21:30:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:21 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.522 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:21.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:21.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:21.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:21 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:22.167 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:22 smithi067 ceph-mon[33346]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:22 smithi055 ceph-mon[40587]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:22 smithi055 ceph-mon[29406]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:23.185 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:30:23.185 INFO:teuthology.orchestra.run.smithi055.stdout: "name": "mon.a", 2024-03-26T21:30:23.185 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster": "ceph", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "admin_socket_mode": "", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_client_required": "cephx, none", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_cluster_required": "cephx", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_debug": "true", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_service_required": "cephx", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "auth_supported": "", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio": "true", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-26T21:30:23.186 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_poll_ms": "250", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_aio_reap_max": "16", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_async_discard_threads": "0", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_block_size": "4096", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio": "false", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_debug_inflight_ios": "false", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_enable_discard": "false", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_flock_retry": "3", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_inject_crash": "0", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-26T21:30:23.187 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring_hipri": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bdev_type": "", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_alloc_size": "1048576", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_allocator": "hybrid", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_buffered_io": "true", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_for_zeros": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_compact_log_sync": "false", 2024-03-26T21:30:23.188 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_max_log_runway": "4194304", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_max_prefetch": "1048576", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_min_flush_size": "524288", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_min_log_runway": "1048576", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_replay_recovery": "false", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_sync_write": "false", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-26T21:30:23.189 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_allocation_from_file": "true", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_allocator": "hybrid", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_create": "true", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_create": "false", 2024-03-26T21:30:23.190 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_path": "", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_db_size": "0", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_path": "", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_preallocate_file": "false", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_size": "107374182400", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_create": "false", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_path": "", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_block_wal_size": "100663296", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs": "true", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T21:30:23.191 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_autotune": "true", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size": "0", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_type": "2q", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_clone_cow": "true", 2024-03-26T21:30:23.192 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_mode": "none", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_csum_type": "crc32c", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_freelist": "false", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-26T21:30:23.193 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_small_allocations": "0", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-26T21:30:23.194 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_default_buffered_read": "true", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_default_buffered_write": "false", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fail_eio": "false", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-26T21:30:23.195 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mount": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_umount": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-26T21:30:23.196 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_ignore_data_csum": "false", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_log_op_age": "5.000000", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_alloc_size": "0", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size": "0", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_max_deferred_txc": "32", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size": "0", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-26T21:30:23.197 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_nid_prealloc": "1024", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_retry_disk_reads": "3", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_cf": "true", 2024-03-26T21:30:23.198 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_spdk_mem": "512", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-26T21:30:23.199 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_tracing": "false", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-26T21:30:23.200 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_zero_block_detection": "false", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-26T21:30:23.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_cluster_require_signatures": "false", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_cluster_require_version": "2", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_require_signatures": "false", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_require_version": "2", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_service_require_signatures": "false", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_service_require_version": "2", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "cephx_sign_messages": "true", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "chdir": "", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_acl_type": "", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_asio_thread_count": "2", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_cache_mid": "0.750000", 2024-03-26T21:30:23.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_cache_size": "16384", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_caps_release_delay": "5", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_check_pool_perm": "true", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_force_sync_read": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_getattr_caps": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_debug_inject_tick_delay": "0", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_die_on_failed_remount": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_dirsize_rbytes": "true", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_force_lazyio": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_fs": "", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_inject_release_failure": "false", 2024-03-26T21:30:23.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_max_inline_size": "4096", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mds_namespace": "", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_metadata": "", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_gid": "-1", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_timeout": "300", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mount_uid": "-1", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_mountpoint": "/", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_notify_timeout": "10", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc": "true", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_dirty": "104857600", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_max_objects": "1000", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_size": "209715200", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_oc_target_dirty": "8388608", 2024-03-26T21:30:23.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_permissions": "true", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_quota": "true", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_quota_df": "true", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_max_bytes": "0", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_max_periods": "4", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_readahead_min": "131072", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_reconnect_stale": "false", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_shutdown_timeout": "30", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_snapdir": ".snap", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_tick_interval": "1", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_trace": "", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_try_dentry_invalidate": "false", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_use_faked_inos": "false", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_use_random_mds": "false", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog": "false", 2024-03-26T21:30:23.205 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_graylog_port": "12201", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_monitors": "default=true", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog": "false", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "clog_to_syslog_level": "info", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "-", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_network": "", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_network_interface": "", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_isal": "false", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_level": "5", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zlib_winsize": "-15", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "compressor_zstd_level": "1", 2024-03-26T21:30:23.206 INFO:teuthology.orchestra.run.smithi055.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_alien_op_num_threads": "6", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crimson_seastar_cpu_cores": "", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location": "", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location_hook": "", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_location_hook_timeout": "10", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "daemonize": "false", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "daos_pool": "tank", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-26T21:30:23.207 INFO:teuthology.orchestra.run.smithi055.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_alienstore": "0/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_allow_any_pool_priority": "false", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asok": "1/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asok_assert_abort": "false", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_asserts_on_shutdown": "true", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_auth": "1/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bdev": "1/3", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bluefs": "1/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_bluestore": "1/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_buffer": "0/1", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_ceph_exporter": "1/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cephfs_mirror": "0/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cephsqlite": "0/5", 2024-03-26T21:30:23.208 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_client": "0/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_compressor": "1/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_context": "0/1", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crimson_interrupt": "0/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crush": "1/1", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_crypto": "1/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_cyanstore": "0/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_deliberately_leak_memory": "false", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_disable_randomized_ping": "false", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_dpdk": "1/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_eventtrace": "1/5", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_filer": "0/1", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_filestore": "1/3", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_finisher": "1/1", 2024-03-26T21:30:23.209 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_fuse": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_heartbeat_testing_span": "0", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_heartbeatmap": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_javaclient": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_journal": "1/3", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_journaler": "0/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_kstore": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_lockdep": "0/1", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mclock": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_balancer": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_locker": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_log": "1/5", 2024-03-26T21:30:23.210 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_log_expire": "1/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_migrator": "1/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mds_quiesce": "3/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_memstore": "1/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mgr": "2/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mgrc": "1/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_mon": "20/20", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_monc": "0/10", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_ms": "1/1", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_none": "0/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objclass": "0/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objectcacher": "0/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_objecter": "0/1", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_optracker": "0/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_osd": "1/5", 2024-03-26T21:30:23.211 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_paxos": "20/20", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_perfcounter": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_prioritycache": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rados": "0/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd": "0/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_mirror": "0/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_pwl": "0/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rbd_replay": "0/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_refs": "0/0", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_reserver": "1/1", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_access": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_datacache": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_dbstore": "1/5", 2024-03-26T21:30:23.212 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_flight": "1/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_notification": "1/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rgw_sync": "1/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_rocksdb": "4/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_backref": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_cache": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_cleaner": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_device": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_epm": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_journal": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_lba": "0/5", 2024-03-26T21:30:23.213 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_odata": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_omap": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_onode": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_t": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_seastore_tm": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_striper": "0/1", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_test": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_throttle": "1/1", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_timer": "0/1", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_tp": "0/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "debug_trace": "1/5", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "device_failure_prediction_mode": "none", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-26T21:30:23.214 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_graylog": "false", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_journald": "false", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_stderr": "true", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "err_to_syslog": "false", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event_tracing": "false", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_addr": "0.0.0.0", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_http_port": "9926", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_prio_limit": "5", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_sort_metrics": "true", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exporter_stats_period": "5", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "fake_statfs_for_testing": "0", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "fatal_signal_handlers": "true", 2024-03-26T21:30:23.215 INFO:teuthology.orchestra.run.smithi055.stdout: "filer_max_purge_ops": "10", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filer_max_truncate_ops": "128", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_apply_finisher_threads": "1", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_blackhole": "false", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_btrfs_clone_range": "true", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_btrfs_snap": "true", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_caller_concurrency": "10", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_collect_device_partition_information": "true", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_commit_timeout": "600.000000", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_inject_read_err": "false", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_omap_check": "false", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_debug_verify_split": "false", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_dump_file": "", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-26T21:30:23.216 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fadvise": "true", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fail_eio": "true", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fd_cache_shards": "16", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fd_cache_size": "128", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fiemap": "false", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fiemap_threshold": "4096", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_inject_stall": "0", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_parallel": "false", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_trailing": "false", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_journal_writeahead": "false", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_kill_at": "0", 2024-03-26T21:30:23.217 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs": "0", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size": "0", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-26T21:30:23.218 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_merge_threshold": "-10", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_odsync_write": "false", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_backend": "rocksdb", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_backend_path": "", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_thread_timeout": "60", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_op_threads": "2", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_punch_hole": "false", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-26T21:30:23.219 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_queue_max_ops": "50", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_seek_data_hole": "false", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_sloppy_crc": "false", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_splice": "false", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_split_multiple": "2", 2024-03-26T21:30:23.220 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_split_rand_factor": "20", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_update_to": "1000", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_enable": "true", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-26T21:30:23.221 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_xfs_extsize": "false", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "filestore_zfs_snap": "false", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fio_dir": "/tmp/fio", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_allow_other": "true", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_atomic_o_trunc": "true", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_big_writes": "true", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_debug": "false", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_default_permissions": "false", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_disable_pagecache": "false", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_max_write": "0", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_multithreaded": "true", 2024-03-26T21:30:23.222 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_require_active_mds": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_set_user_groups": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_move": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_read": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_splice_write": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "fuse_use_invalidate_cb": "true", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "gss_target_name": "ceph", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_file": "", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_inject_failure": "0", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_interval": "5", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "host": "smithi055", 2024-03-26T21:30:23.223 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "inject_early_sigterm": "false", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "jaeger_agent_port": "6799", 2024-03-26T21:30:23.224 INFO:teuthology.orchestra.run.smithi055.stdout: "jaeger_tracing_enable": "false", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_aio": "true", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_align_min_size": "65536", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_block_align": "true", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_block_size": "4096", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_dio": "true", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_discard": "false", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_force_aio": "false", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_ignore_corruption": "false", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_max_write_bytes": "10485760", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_max_write_entries": "100", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_replay_from": "0", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-26T21:30:23.225 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_write_header_frequency": "0", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journal_zero_on_create": "false", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_prefetch_periods": "10", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_prezero_periods": "5", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "journaler_write_head_interval": "15", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "key": "", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "keyfile": "", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_backend": "rocksdb", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_default_stripe_size": "65536", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_fsck_on_mount": "false", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_max_bytes": "67108864", 2024-03-26T21:30:23.226 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_max_ops": "512", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_nid_prealloc": "1024", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_onode_map_size": "1024", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_sync_submit_transaction": "false", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "kstore_sync_transaction": "false", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "librados_thread_count": "2", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "lockdep": "false", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "lockdep_force_backtrace": "false", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_coarse_timestamps": "true", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_flush_on_exit": "false", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_graylog_host": "127.0.0.1", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_graylog_port": "12201", 2024-03-26T21:30:23.227 INFO:teuthology.orchestra.run.smithi055.stdout: "log_max_new": "1000", 2024-03-26T21:30:23.228 INFO:teuthology.orchestra.run.smithi055.stdout: "log_max_recent": "10000", 2024-03-26T21:30:23.228 INFO:teuthology.orchestra.run.smithi055.stdout: "log_stderr_prefix": "", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_stop_at_utilization": "0.970000", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_file": "true", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_graylog": "false", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_journald": "false", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_stderr": "false", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "log_to_syslog": "false", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "max_rotating_auth_attempts": "10", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_action_on_write_error": "1", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_alternate_name_max": "8192", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_asio_thread_count": "2", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_export_pin": "true", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_dirs": "true", 2024-03-26T21:30:23.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_interval": "5", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_interval": "10", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_max": "-1", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_max_until": "-1", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_merge_size": "50", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_midchunk": "0.300000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_min_start": "0.200000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_minchunk": "0.001000", 2024-03-26T21:30:23.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_mode": "0", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_need_max": "1.200000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_need_min": "0.800000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_overload_epochs": "2", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_bits": "3", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_size": "10000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_target_decay": "10.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-26T21:30:23.231 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_grace": "15.000000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_interval": "4.000000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_mid": "0.700000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_quiesce_delay": "0", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_quiesce_sleep": "200", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_quiesce_splitauth": "true", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_quiesce_threshold": "524288", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_release_free_interval": "10", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_reservation": "0.050000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_interval": "1", 2024-03-26T21:30:23.232 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache_trim_threshold": "262144", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_prealloc_inos": "1000", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_connect_bootstrapping": "false", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_damage_table_max_entries": "10000", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_auth_pins": "false", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_frag": "false", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_scatterstat": "false", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_debug_subtrees": "false", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_decay_halflife": "5.000000", 2024-03-26T21:30:23.233 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_default_dir_hash": "2", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_defer_session_stale": "true", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_deny_all_reconnect": "false", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_keys_per_op": "16384", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_max_commit_size": "10", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_max_entries": "0", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dir_prefetch": "true", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_on_map": "false", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_early_reply": "true", 2024-03-26T21:30:23.234 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_enable_op_tracker": "true", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_enforce_unique_name": "true", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_random": "true", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_health_summarize_threshold": "10", 2024-03-26T21:30:23.235 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_health_dummy": "false", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_migrator_session_race": "false", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_join_fs": "", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_journal_format": "1", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_create_at": "0", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_export_at": "0", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_import_at": "0", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_at": "0", 2024-03-26T21:30:23.236 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_expire_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_journal_replay_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_link_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_mdstable_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_openc_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_rename_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_shutdown_at": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_event_large_threshold": "524288", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_events_per_segment": "1024", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_max_events": "-1", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_max_segments": "128", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_pause": "false", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_segment_size": "0", 2024-03-26T21:30:23.237 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_trim_threshold": "128", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log_warn_factor": "2.000000", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_caps_per_client": "1048576", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_completed_flushes": "100000", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_completed_requests": "100000", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_export_size": "20971520", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_file_recover": "32", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_files": "64", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_ops": "8192", 2024-03-26T21:30:23.238 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_max_snaps_per_dir": "100", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_metrics_update_interval": "2", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_min_caps_per_client": "100", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_min_caps_working_set": "10000", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_numa_node": "-1", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_complaint_time": "30.000000", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_duration": "600", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_size": "20", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_slow_op_size": "20", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-26T21:30:23.239 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_op_log_threshold": "5", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_ping_grace": "15", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_ping_interval": "5", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_caps": "30000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_recall_warning_threshold": "262144", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_replay_interval": "1.000000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_root_ino_gid": "0", 2024-03-26T21:30:23.240 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_root_ino_uid": "0", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_shutdown_check": "0", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_skip_ino": "0", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-26T21:30:23.241 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_max_uid": "4294967294", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_min_uid": "0", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_snap_rstat": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_standby_replay_damaged": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_symlink_recovery": "true", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_thrash_exports": "0", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_thrash_fragments": "0", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_tick_interval": "5.000000", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_valgrind_exit": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_verify_backtrace": "1", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_verify_scatter": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_wipe_sessions": "false", 2024-03-26T21:30:23.242 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool_debug": "false", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_device_bytes": "1073741824", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_page_set": "false", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "memstore_page_size": "65536", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_bytes": "134217728", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_messages": "512", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_disabled_modules": "", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_max_pg_num_change": "128", 2024-03-26T21:30:23.243 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mds_bytes": "134217728", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mds_messages": "128", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mon_bytes": "134217728", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_mon_messages": "128", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_osd_bytes": "536870912", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_osd_messages": "8192", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_pool": "true", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_standby_modules": "true", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_stats_period": "5", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_stats_threshold": "5", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_tick_period": "2", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-26T21:30:23.244 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_allow_pool_delete": "true", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_allow_pool_size_one": "false", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_auth_validate_all_caps": "true", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_bytes": "104857600", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_directed_command_retry": "2", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_hunt_parallel": "3", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_log_interval": "1.000000", 2024-03-26T21:30:23.245 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_ping_interval": "10.000000", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_client_target_rank": "-1", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_file_level": "debug", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_file": "true", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_journald": "true", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-26T21:30:23.246 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_bootstrap": "false", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_start": "false", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_compact_on_trim": "true", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_cpu_threads": "4", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_crush_min_required_version": "hammer", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_daemon_bytes": "419430400", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-26T21:30:23.247 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_avail_crit": "5", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_avail_warn": "5", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_data_size_warn": "16106127360", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_json": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_dump_transactions": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_extra_checks": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_reef": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_no_require_squid": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-26T21:30:23.248 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_down_mkfs_grace": "60", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_down_uptime_grace": "60", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_election_default_strategy": "1", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_election_timeout": "5.000000", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ping_divisor": "2", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_enable_op_tracker": "true", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_fake_pool_delete": "false", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_force_quorum_join": "false", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_globalid_prealloc": "10000", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_detail_to_clog": "true", 2024-03-26T21:30:23.249 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_log_update_period": "5", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_max_detail": "50", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog": "true", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog_interval": "600", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_host": "[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_host_override": "", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_initial_members": "", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease": "5.000000", 2024-03-26T21:30:23.250 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_full_interval": "50", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_max": "10000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_log_max_summary": "50", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_log_epochs": "500", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_osd": "10000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_pg_per_osd": "10000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_pool_pg_num": "65536", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-26T21:30:23.251 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_force_trim_to": "0", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mds_skip_sanity": "false", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_memory_autotune": "true", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_memory_target": "2147483648", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_beacon_grace": "30", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_digest_period": "5", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_inactive_grace": "60", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_min_osdmap_epochs": "50", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_complaint_time": "30", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_duration": "600", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_size": "20", 2024-03-26T21:30:23.252 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_slow_op_size": "20", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_op_log_threshold": "5", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_in": "false", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_cache_size": "500", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-26T21:30:23.253 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_down_out_interval": "600", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_force_trim_to": "0", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_down_reporters": "2", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-26T21:30:23.254 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_report_timeout": "900", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-26T21:30:23.255 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_stuck_threshold": "60", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_probe_timeout": "2.000000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_max_change": "0.050000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_max_osds": "4", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-26T21:30:23.256 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_interval": "86400", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_max_keys": "100", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_scrub_timeout": "300", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_session_timeout": "300", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_smart_report_timeout": "5", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stat_smooth_intervals": "6", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_pool_min_size": "2", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_pool_size": "4", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_debug": "false", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-26T21:30:23.257 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_provider_kill_at": "0", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_requester_kill_at": "0", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_sync_timeout": "60.000000", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_target_pg_per_osd": "100", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_tick_interval": "5", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_timecheck_interval": "300.000000", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_use_min_delay_socket": "false", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_older_version_delay": "604800", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-26T21:30:23.258 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_misplaced": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_older_version": "true", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-26T21:30:23.259 INFO:teuthology.orchestra.run.smithi055.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "monmap": "", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "motr_tracing_enabled": "false", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_op_threads": "3", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_cm": "false", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_device_name": "", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_dscp": "96", 2024-03-26T21:30:23.260 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_local_gid": "", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_port_num": "1", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_sl": "3", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_support_srq": "true", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_rdma_type": "ib", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_async_reap_threshold": "5", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_before_connect": "false", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_ipv4": "true", 2024-03-26T21:30:23.261 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_ipv6": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_msgr1": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_msgr2": "true", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_port_max": "7568", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_port_min": "6800", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_retry_count": "3", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_bind_retry_delay": "5", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_client": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mds": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mgr": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_mon": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_blackhole_osd": "false", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_client_mode": "crc secure", 2024-03-26T21:30:23.262 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_cluster_mode": "crc secure", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_cluster_type": "", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_compress_secure": "false", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_connection_idle_timeout": "900", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_connection_ready_timeout": "10", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_crc_data": "true", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_crc_header": "true", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_bad_msg": "false", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_bug": "true", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_old_message": "true", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_skipped_message": "false", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_coremask": "0xF", 2024-03-26T21:30:23.263 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_enable_tso": "true", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hugepages": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_lro": "true", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_memory_channel": "4", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_pmd": "", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_port_id": "0", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-26T21:30:23.264 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_dump_on_send": "false", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_initial_backoff": "0.200000", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_max": "1.000000", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_delay_type": "", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_network_congestion": "0", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_inject_socket_failures": "0", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_learn_addr_from_peer": "true", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_max_accept_failures": "4", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_max_backoff": "15.000000", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_client_mode": "secure crc", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-26T21:30:23.265 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_mon_service_mode": "secure crc", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compress_min_size": "1024", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compress_mode": "none", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_pq_min_cost": "65536", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_public_type": "", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_service_mode": "crc secure", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_listen_backlog": "512", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_nodelay": "true", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_tcp_rcvbuf": "0", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "ms_type": "async+posix", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "no_config_file": "false", 2024-03-26T21:30:23.266 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_completion_locks_per_session": "32", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inflight_ops": "1024", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_tick_interval": "5.000000", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter_timeout": "10.000000", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objectstore_blackhole": "false", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "openssl_engine_opts": "", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_delay_time": "5.000000", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_hist_halflife": "1000", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_max_low_ops": "2", 2024-03-26T21:30:23.267 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_max_ops": "4", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_agent_slop": "0.020000", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_asio_thread_count": "2", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_async_recovery_min_cost": "100", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_scan_max": "512", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backfill_scan_min": "64", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_degraded": "false", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_peering": "false", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_backoff_on_unfound": "true", 2024-03-26T21:30:23.268 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_beacon_report_interval": "300", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_duration": "30", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_max_block_size": "67108864", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_blkin_trace_all": "false", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_check_for_log_corruption": "false", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-26T21:30:23.269 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_class_update_on_start": "true", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_message_cap": "256", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_message_size_cap": "524288000", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_op_priority": "63", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_client_watch_timeout": "30", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_command_max_records": "256", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_compact_on_start": "false", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_update_on_start": "true", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_crush_update_weight_set": "true", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-26T21:30:23.270 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_feed_pullee": "-1", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_misdirected_ops": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_no_acting_change": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_no_purge_strays": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_op_order": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-26T21:30:23.271 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_shutdown": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_trim_objects": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_snaps": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_keys": "1024", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-26T21:30:23.272 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_stride": "524288", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_default_notify_timeout": "30", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep": "0.000000", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_discard_disconnected_ops": "true", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_enable_op_tracker": "true", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-26T21:30:23.273 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_info": "true", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown": "true", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_function_tracing": "false", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_grace": "20", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_interval": "6", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-26T21:30:23.274 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_peers": "10", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_min_size": "2000", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_stale": "600", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_max_size": "100000", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_min_size": "1000", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_journal_size": "5120", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_kill_backfill_at": "0", 2024-03-26T21:30:23.275 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_cache_size": "50", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_dedup": "true", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_message_max": "40", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_map_share_max_epochs": "40", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_attr_name_len": "100", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_attr_size": "0", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_backfills": "1", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_markdown_count": "5", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_markdown_period": "600", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_name_len": "2048", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_namespace_len": "256", 2024-03-26T21:30:23.276 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_object_size": "134217728", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_blocked_by": "16", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_log_entries": "10000", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_pgls": "1024", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_push_cost": "8388608", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_push_objects": "10", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_scrubs": "3", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_trimming_pgs": "2", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_write_op_reply_len": "64", 2024-03-26T21:30:23.277 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_max_write_size": "90", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_profile": "balanced", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-26T21:30:23.278 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_base": "805306368", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_cache_min": "134217728", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target": "4294967296", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target_autotune": "false", 2024-03-26T21:30:23.279 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_min_pg_log_entries": "250", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_min_recovery_priority": "0", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_report_interval": "5", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_num_cache_shards": "32", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_num_op_tracker_shard": "32", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_auto_affinity": "true", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_node": "-1", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_numa_prefer_iface": "true", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-26T21:30:23.280 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objecter_finishers": "1", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore": "bluestore", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore_fuse": "false", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_objectstore_tracing": "false", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_complaint_time": "30.000000", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_duration": "600", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_size": "20", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_slow_op_size": "20", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_log_threshold": "5", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards": "0", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards_hdd": "5", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_shards_ssd": "8", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-26T21:30:23.281 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_pq_min_cost": "65536", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_queue_cut_off": "high", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_op_thread_timeout": "15", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_open_classes_on_start": "true", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_os_flags": "0", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_peering_op_priority": "255", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_delete_cost": "1048576", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_delete_priority": "5", 2024-03-26T21:30:23.282 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_trim_max": "10000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_log_trim_min": "100", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-26T21:30:23.283 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_crimson": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_flags": "0", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_min_size": "0", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-26T21:30:23.284 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pg_num": "32", 2024-03-26T21:30:23.285 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_pgp_num": "0", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_read_ratio": "70", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_size": "3", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_default_type": "replicated", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_push_per_object_cost": "1000", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recover_clone_overlap": "true", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_cost": "20971520", 2024-03-26T21:30:23.286 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active": "0", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_max_single_start": "1", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_op_priority": "3", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_priority": "5", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep": "0.000000", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-26T21:30:23.287 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_repair_during_recovery": "false", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_requested_scrub_priority": "120", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_auto_repair": "false", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_begin_hour": "0", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_begin_week_day": "0", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_chunk_max": "25", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_chunk_min": "5", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_cost": "52428800", 2024-03-26T21:30:23.288 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_during_recovery": "false", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_end_hour": "0", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_end_week_day": "0", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_event_cost": "4096", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_invalid_stats": "true", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_max_preemptions": "5", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_priority": "5", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-26T21:30:23.289 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_scrub_sleep": "0.000000", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_skip_data_digest": "false", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_smart_report_timeout": "5", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_cost": "1048576", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_priority": "5", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-26T21:30:23.290 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_target_transaction_size": "30", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-26T21:30:23.291 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_tracing": "false", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_use_stale_snap": "false", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "osdc_blkin_trace_all": "false", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_kill_at": "0", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_max_join_drift": "10", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_min": "500", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_min_wait": "0.050000", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_propose_interval": "1.000000", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_max": "500", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_service_trim_min": "10", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_stash_full_interval": "25", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_trim_max": "500", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "paxos_trim_min": "250", 2024-03-26T21:30:23.292 INFO:teuthology.orchestra.run.smithi055.stdout: "perf": "true", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "pid_file": "", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "-", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrv": "", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "public_bind_addr": "-", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "public_network": "", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "public_network_interface": "", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "qat_compressor_enabled": "false", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "qat_compressor_session_max_number": "256", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_mon_op_timeout": "0", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_osd_op_timeout": "0", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "rados_tracing": "false", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_atime_update_interval": "60", 2024-03-26T21:30:23.293 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_balance_parent_reads": "false", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_balance_snap_reads": "false", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blkin_trace_all": "false", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache": "true", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_policy": "writearound", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_size": "33554432", 2024-03-26T21:30:23.294 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_clone_copy_on_read": "false", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_compression_hint": "none", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_concurrent_management_ops": "10", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_clone_format": "auto", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_data_pool": "", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_features": "61", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_format": "2", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_map_options": "", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_order": "22", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_pool": "rbd", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-26T21:30:23.295 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_stripe_count": "0", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_default_stripe_unit": "0", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_enable_alloc_hint": "true", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_io_scheduler": "simple", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-26T21:30:23.296 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_order": "24", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_pool": "", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_journal_splay_width": "4", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_localize_parent_reads": "false", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_localize_snap_reads": "false", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-26T21:30:23.297 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-26T21:30:23.298 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-26T21:30:23.299 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_mtime_update_interval": "60", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_non_blocking_aio": "true", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_op_thread_timeout": "60", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_op_threads": "1", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_parent_cache_enabled": "false", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_plugins": "", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_burst": "0", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_bps_limit": "0", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_exclude_ops": "0", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_burst": "0", 2024-03-26T21:30:23.300 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_iops_limit": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-26T21:30:23.301 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_read_from_replica_policy": "default", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_skip_partial_discard": "true", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_tracing": "false", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_validate_names": "true", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd_validate_pool": "true", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "restapi_base_url": "", 2024-03-26T21:30:23.302 INFO:teuthology.orchestra.run.smithi055.stdout: "restapi_log_level": "", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_acl_grants_max_num": "100", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_admin_entry": "admin", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_backend_store": "rados", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_barbican_url": "", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_beast_enable_async": "true", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_counters_cache": "false", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-26T21:30:23.303 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_enabled": "true", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_expiry_interval": "900", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cache_lru_size": "10000", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_config_store": "rados", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_content_length_compat": "false", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_copy_obj_progress": "true", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cors_rules_max_num": "100", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_cross_domain_policy": "", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_addr": "", 2024-03-26T21:30:23.304 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_password": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_kmip_username": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_require_ssl": "true", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-26T21:30:23.305 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_addr": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_auth": "token", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_namespace": "", 2024-03-26T21:30:23.306 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_prefix": "", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_token_file": "", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_buffersize": "524288", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_low_speed_time": "300", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-26T21:30:23.307 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_d4n_port": "6379", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_changes_size": "1000", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_num_shards": "128", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_log_window": "30", 2024-03-26T21:30:23.308 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-26T21:30:23.309 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dns_name": "", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dns_s3website_name": "", 2024-03-26T21:30:23.310 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_dynamic_resharding": "true", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_gc_threads": "true", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_lc_threads": "true", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_ops_log": "false", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_quota_threads": "true", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_static_website": "false", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enable_usage_log": "false", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_enforce_swift_acls": "true", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_exit_timeout_secs": "120", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_expose_bucket": "false", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_extended_http_attrs": "", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_filter": "none", 2024-03-26T21:30:23.311 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_frontends": "beast port=7480", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_deferred": "50", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_objs": "32", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_gc_processor_period": "3600", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-26T21:30:23.312 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_init_timeout": "300", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_kafka_connection_idle": "300", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_domain": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_password": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_password_path": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_project": "", 2024-03-26T21:30:23.313 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_tenant": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_token": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_token_path": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_admin_user": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_api_version": "2", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_domain": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_password": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_project": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_barbican_user": "", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-26T21:30:23.314 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_url": "", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_debug_interval": "-1", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_lock_max_time": "90", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_objs": "32", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_rules": "1000", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_worker": "3", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lc_thread_delay": "0", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_dnattr": "uid", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-26T21:30:23.315 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_searchfilter": "", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_http_headers": "", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_log_object_name_utc": "false", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attr_name_len": "0", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attr_size": "0", 2024-03-26T21:30:23.316 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_chunk_size": "4194304", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_listing_results": "1000", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_notify_retries": "10", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_put_param_size": "1048576", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_put_size": "5368709120", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_max_slo_entries": "1000", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_md_log_max_shards": "64", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-26T21:30:23.317 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_mp_lock_max_time": "600", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_max_gc": "300", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-26T21:30:23.318 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_num_async_rados_threads": "32", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_num_control_oids": "8", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_numa_node": "-1", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_chunk_size": "100", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_gc_interval": "600", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-26T21:30:23.319 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_thread_timeout": "600", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_op_tracing": "false", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_token": "", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_url": "", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_opa_verify_ssl": "true", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_rados": "false", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_ops_log_socket_path": "", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_push_interval": "2.000000", 2024-03-26T21:30:23.320 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_lanes": "3", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_cache_partitions": "3", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_print_continue": "true", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-26T21:30:23.321 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rados_tracing": "false", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm": "", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm_id": "", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_region": "", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_request_uri": "", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_batch_size": "64", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-26T21:30:23.322 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_max_aio": "128", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_num_logs": "16", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_reshard_thread_interval": "600", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_resolve_cname": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_run_sync_thread": "true", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-26T21:30:23.323 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_scheduler_type": "throttler", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_script_uri": "", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_service_provider_name": "", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_client_id": "", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_client_secret": "", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_entry": "sts", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_key": "sts", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_min_session_duration": "900", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sts_token_introspection_url": "", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_account_in_url": "false", 2024-03-26T21:30:23.324 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_auth_entry": "auth", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_auth_url": "", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_custom_header": "", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_need_stats": "true", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_tenant_name": "", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_token_expiration": "86400", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_url": "", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_url_prefix": "swift", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_lease_period": "120", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-26T21:30:23.325 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_thread_pool_size": "512", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_comment": "", 2024-03-26T21:30:23.326 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_createby": "", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_encoding": "", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_flag": "false", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_origin": "", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_torrent_tracker": "", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_trust_forwarded_https": "false", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_max_shards": "32", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_usage_max_user_shards": "1", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_use_opa_authz": "false", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_counters_cache": "false", 2024-03-26T21:30:23.327 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_max_buckets": "1000", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_policies_max_num": "100", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_user_unique_email": "true", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_verify_ssl": "true", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone": "", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone_id": "", 2024-03-26T21:30:23.328 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup": "", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup_id": "", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rgwlc_auto_session_clear": "true", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_block_size": "4096", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_size": "536870912", 2024-03-26T21:30:23.329 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_index_type": "binary_search", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_partition_filters": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_perf": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-26T21:30:23.330 INFO:teuthology.orchestra.run.smithi055.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "run_dir": "/var/run/ceph", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_block_create": "true", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_cache_lru_size": "67108864", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_default_max_object_size": "16777216", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_device_size": "53687091200", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_capacity": "16", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-26T21:30:23.331 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_main_device_type": "SSD", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "seastore_segment_size": "67108864", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "set_keepcaps": "false", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "setgroup": "ceph", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "setuser": "ceph", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "setuser_match_path": "", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "thp": "false", 2024-03-26T21:30:23.332 INFO:teuthology.orchestra.run.smithi055.stdout: "threadpool_default_timeout": "60", 2024-03-26T21:30:23.333 INFO:teuthology.orchestra.run.smithi055.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-26T21:30:23.333 INFO:teuthology.orchestra.run.smithi055.stdout: "throttler_perf_counter": "true" 2024-03-26T21:30:23.333 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:23 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:23 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:30:23.500 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:23 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:30:23.501 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:23 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:30:23.662 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:23 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:30:23.662 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:23 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:30:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:24 smithi067 ceph-mon[33346]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:24 smithi067 ceph-mon[33346]: 5.1b deep-scrub starts 2024-03-26T21:30:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:24 smithi067 ceph-mon[33346]: 5.1b deep-scrub ok 2024-03-26T21:30:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:24 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:30:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:24 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:30:24.688 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[29406]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:24.688 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[29406]: 5.1b deep-scrub starts 2024-03-26T21:30:24.688 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[29406]: 5.1b deep-scrub ok 2024-03-26T21:30:24.688 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:30:24.688 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:30:24.689 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[40587]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:24.689 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[40587]: 5.1b deep-scrub starts 2024-03-26T21:30:24.689 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[40587]: 5.1b deep-scrub ok 2024-03-26T21:30:24.689 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:30:24.689 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:24 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:30:24.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-26T21:30:25.500 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.888 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:25.889 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:30:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:30:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:26.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:26.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:30:26.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:30:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:30:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:30:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:30:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:26 smithi067 ceph-mon[33346]: Reconfiguring daemon alertmanager.a on smithi055 2024-03-26T21:30:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:30:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:30:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[40587]: Reconfiguring daemon alertmanager.a on smithi055 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T21:30:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:26 smithi055 ceph-mon[29406]: Reconfiguring daemon alertmanager.a on smithi055 2024-03-26T21:30:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:27 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:30:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:27 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:30:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:27.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:30:27.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:30:27.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:27.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:30:27.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:30:27.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:28.986 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:28 smithi055 systemd[1]: Stopping Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:28.986 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[40587]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:28.986 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[40587]: 6.15 deep-scrub starts 2024-03-26T21:30:28.987 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[40587]: 6.15 deep-scrub ok 2024-03-26T21:30:28.987 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[29406]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:28.987 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[29406]: 6.15 deep-scrub starts 2024-03-26T21:30:28.987 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-mon[29406]: 6.15 deep-scrub ok 2024-03-26T21:30:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:28 smithi067 ceph-mon[33346]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:30:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:28 smithi067 ceph-mon[33346]: 6.15 deep-scrub starts 2024-03-26T21:30:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:28 smithi067 ceph-mon[33346]: 6.15 deep-scrub ok 2024-03-26T21:30:29.248 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:30:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stdout:mon.a: {} 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stdout:mon.b: {} 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stdout:mon.c: {} 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:30:29.719 INFO:teuthology.orchestra.run.smithi055.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:30:29.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:30:29.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:30:29.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3193826023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:30:29.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[75235]: ts=2024-03-26T21:30:29.602Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:30:29.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:30:29.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:30:29.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3193826023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:30:29.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:29 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:30:29.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:29 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:30:29.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3193826023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:30:30.284 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-26T21:30:30.294 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T21:30:30.295 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T21:30:30.295 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-26T21:30:30.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-26T21:30:30.296 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-26T21:30:30.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T21:30:30.296 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:30:30.297 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T21:30:30.297 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T21:30:30.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4 2024-03-26T21:30:30.297 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-26T21:30:30.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:30.298 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-26T21:30:30.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:30:30.298 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-26T21:30:30.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:30:30.299 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-26T21:30:30.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:30:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:30.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:30.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.456 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:30.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:30.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:30.565 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:30 smithi055 podman[77620]: 2024-03-26 21:30:30.35247173 +0000 UTC m=+0.965763319 container died e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:30.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:30.616 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:30 smithi067 systemd[1]: Stopping Ceph osd.4 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:30.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.709 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:30.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:30.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.834 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:30.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:30.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:30.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:30.962 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:30 smithi055 podman[77620]: 2024-03-26 21:30:30.776818438 +0000 UTC m=+1.390110038 container cleanup e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:30.962 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:30 smithi055 bash[77620]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T21:30:30.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[29406]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T21:30:30.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:30.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[29406]: osd.4 marked itself down and dead 2024-03-26T21:30:30.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[40587]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T21:30:30.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:30.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:30 smithi055 ceph-mon[40587]: osd.4 marked itself down and dead 2024-03-26T21:30:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-mon[33346]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T21:30:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-mon[33346]: osd.4 marked itself down and dead 2024-03-26T21:30:30.989 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[38347]: 2024-03-26T21:30:30.622+0000 7f66ed126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:30:30.990 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[38347]: 2024-03-26T21:30:30.622+0000 7f66ed126700 -1 osd.4 92 *** Got signal Terminated *** 2024-03-26T21:30:30.990 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:30 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[38347]: 2024-03-26T21:30:30.622+0000 7f66ed126700 -1 osd.4 92 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:30:31.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.087 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:31.160 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:31.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:31.289 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:30 smithi055 podman[77659]: 2024-03-26 21:30:30.970139224 +0000 UTC m=+1.340846543 container remove e42cd35e1d6c13e07eca14a1733399bca07ec6a5b66d83304ef41320bd07beaf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:31.289 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:31 smithi055 podman[77659]: 2024-03-26 21:30:31.054391126 +0000 UTC m=+1.425098447 volume remove d42ca676da363967b28f04c62f3a04ad96774eeae42349321c88bb5b7678b5e9 2024-03-26T21:30:31.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:31.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:31.408 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:31 smithi067 podman[65427]: 2024-03-26 21:30:31.123550754 +0000 UTC m=+0.618962329 container died f8e4ae69c002f3c90417d1375a0d27106999508d53d2977c3e541635ae18998b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True) 2024-03-26T21:30:31.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:31.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:31.595 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:31.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:31.721 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:31.739 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:31 smithi067 podman[65427]: 2024-03-26 21:30:31.50646786 +0000 UTC m=+1.001879433 container cleanup f8e4ae69c002f3c90417d1375a0d27106999508d53d2977c3e541635ae18998b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T21:30:31.739 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:31 smithi067 bash[65427]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4 2024-03-26T21:30:31.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: 6.13 deep-scrub starts 2024-03-26T21:30:31.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: 6.13 deep-scrub ok 2024-03-26T21:30:31.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: osdmap e93: 8 total, 7 up, 8 in 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[40587]: osdmap e94: 8 total, 7 up, 8 in 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: 6.13 deep-scrub starts 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: 6.13 deep-scrub ok 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: osdmap e93: 8 total, 7 up, 8 in 2024-03-26T21:30:31.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:31 smithi055 ceph-mon[29406]: osdmap e94: 8 total, 7 up, 8 in 2024-03-26T21:30:31.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:31.846 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:31.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:31.972 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:31.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:32.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:32.078 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:32.079 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.099 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:32.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:32.113 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:31 smithi067 podman[65462]: 2024-03-26 21:30:31.782731286 +0000 UTC m=+0.656188215 container remove f8e4ae69c002f3c90417d1375a0d27106999508d53d2977c3e541635ae18998b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T21:30:32.113 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: 6.13 deep-scrub starts 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: 6.13 deep-scrub ok 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: osdmap e93: 8 total, 7 up, 8 in 2024-03-26T21:30:32.114 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:31 smithi067 ceph-mon[33346]: osdmap e94: 8 total, 7 up, 8 in 2024-03-26T21:30:32.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:32.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:32.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:32.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:32.437 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.02399622 +0000 UTC m=+0.018154240 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:30:32.437 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.13710962 +0000 UTC m=+0.131267674 container create cd57f286ae436fbba5e97772f2e616bf422756299d25810ea8c1f7a179bc1a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True) 2024-03-26T21:30:32.437 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.361495923 +0000 UTC m=+0.355653918 container init cd57f286ae436fbba5e97772f2e616bf422756299d25810ea8c1f7a179bc1a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T21:30:32.437 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.364723356 +0000 UTC m=+0.358881360 container start cd57f286ae436fbba5e97772f2e616bf422756299d25810ea8c1f7a179bc1a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T21:30:32.437 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.444768053 +0000 UTC m=+0.438926061 container attach cd57f286ae436fbba5e97772f2e616bf422756299d25810ea8c1f7a179bc1a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:30:32.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:32.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:32.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:32.601 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:32.654 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:32.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:32.735 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:32 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T21:30:32.735 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:32 smithi055 systemd[1]: Stopped Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:32.735 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:32 smithi055 systemd[1]: Starting Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:32.739 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:30:32 smithi067 podman[65574]: 2024-03-26 21:30:32.667293632 +0000 UTC m=+0.661451632 container died cd57f286ae436fbba5e97772f2e616bf422756299d25810ea8c1f7a179bc1a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:30:32.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:32.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:32.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:33.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[29406]: pgmap v52: 132 pgs: 15 peering, 103 active+clean, 14 stale+active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-26T21:30:33.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[29406]: 3.1f deep-scrub starts 2024-03-26T21:30:33.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[29406]: 3.1f deep-scrub ok 2024-03-26T21:30:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:30:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[40587]: pgmap v52: 132 pgs: 15 peering, 103 active+clean, 14 stale+active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-26T21:30:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[40587]: 3.1f deep-scrub starts 2024-03-26T21:30:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[40587]: 3.1f deep-scrub ok 2024-03-26T21:30:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:32 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:30:33.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:33.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:33.132 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:32 smithi067 ceph-mon[33346]: pgmap v52: 132 pgs: 15 peering, 103 active+clean, 14 stale+active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-26T21:30:33.132 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:32 smithi067 ceph-mon[33346]: 3.1f deep-scrub starts 2024-03-26T21:30:33.132 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:32 smithi067 ceph-mon[33346]: 3.1f deep-scrub ok 2024-03-26T21:30:33.132 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:32 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:30:33.207 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:33.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:33.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T21:30:33.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:33.364 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:30:33.365 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T21:30:33.365 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-03-26T21:30:33.365 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi067.front.sepia.ceph.com 2024-03-26T21:30:33.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.4 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-03-26T21:30:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.470 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:33.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:33.565 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T21:30:33.572 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.597 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:33.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:33.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:33.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:33.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:33 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:30:33.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:33 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:30:33.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:33 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:30:33.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:33 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:30:33.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.849 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:33.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:33.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:33.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:33 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:30:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:33 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:30:34.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:34.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:34.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:34.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:34.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.356 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:34.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:34.457 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:34.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:34.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:34.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:34.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:34.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:34.734 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[40587]: pgmap v54: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:30:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:30:34.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[29406]: pgmap v54: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:30:34.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:30:34.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:34 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:30:34.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:34.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:34.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:34.987 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:34.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:34 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:30:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:34 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:30:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:34 smithi067 ceph-mon[33346]: pgmap v54: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T21:30:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:34 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:30:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:34 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:30:35.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:35.113 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:35.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:35.218 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:34 smithi055 podman[78459]: 2024-03-26 21:30:34.808686377 +0000 UTC m=+0.352380142 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T21:30:35.218 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:34 smithi055 podman[78459]: 2024-03-26 21:30:34.933894804 +0000 UTC m=+0.477588545 volume create 1ceaf5ef1c99cb4180a605e2fcc162f926b0664d8682e887bf45f794f1cf177a 2024-03-26T21:30:35.218 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 podman[78459]: 2024-03-26 21:30:35.026235917 +0000 UTC m=+0.569929658 container create a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:35.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:35.365 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:35.446 INFO:teuthology.orchestra.run.smithi067.stdout:6.e 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.7 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.6 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.1e 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.15 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.12 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.f 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.c 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.a 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.9 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.5 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.f 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:6.2 2024-03-26T21:30:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:5.1d 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:5.1a 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.e 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.b 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:5.1c 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.1a 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.a 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:6.1a 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.6 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.1b 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.15 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:6.17 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:5.1f 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:4.5 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:3.13 2024-03-26T21:30:35.448 INFO:teuthology.orchestra.run.smithi067.stdout:4.1b 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.1d 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:5.e 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.11 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.d 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:6.1d 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.14 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.1d 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.f 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.13 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.1 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.e 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:6.14 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.18 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.1c 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:3.19 2024-03-26T21:30:35.449 INFO:teuthology.orchestra.run.smithi067.stdout:4.1a 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.2 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.9 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.1 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:3.9 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.b 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.f 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:3.3 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:5.0 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:4.c 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:5.10 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:5.11 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:5.18 2024-03-26T21:30:35.450 INFO:teuthology.orchestra.run.smithi067.stdout:5.1 2024-03-26T21:30:35.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:35.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:35.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:35.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 podman[78459]: 2024-03-26 21:30:35.362583469 +0000 UTC m=+0.906277209 container init a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 podman[78459]: 2024-03-26 21:30:35.365135957 +0000 UTC m=+0.908829700 container start a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.377Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.378Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.379Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.55 port=9094 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.380Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.408Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.409Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.412Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T21:30:35.816 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:35.412Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T21:30:35.817 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 bash[78459]: a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e 2024-03-26T21:30:35.817 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:35 smithi055 systemd[1]: Started Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:35.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:35.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:35.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:35.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:35.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:36.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:36.118 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:36.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:36.243 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:36.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:36.470 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:36.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:36.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: pgmap v55: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:30:36.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[40587]: Reconfiguring daemon prometheus.a on smithi067 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: pgmap v55: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:30:36.624 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:36 smithi055 ceph-mon[29406]: Reconfiguring daemon prometheus.a on smithi067 2024-03-26T21:30:36.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:36.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: pgmap v55: 132 pgs: 58 peering, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T21:30:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T21:30:36.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:30:36.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:30:36.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:36 smithi067 ceph-mon[33346]: Reconfiguring daemon prometheus.a on smithi067 2024-03-26T21:30:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.750 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:36.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:36.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:36.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:36.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:36.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:37.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:37.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:37.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:37.188 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:30:37.188 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:30:37.188 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 0 2024-03-26T21:30:37.189 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:30:37.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:37.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:37.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:37.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:37.503 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_0' created 2024-03-26T21:30:37.503 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:37 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:37.381Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000821787s 2024-03-26T21:30:37.511 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:37.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.639 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:37.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:37.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T21:30:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T21:30:37.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:37.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:37.896 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T21:30:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:37.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:37.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.4 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 5.5 --file /var/log/ceph/exp.5.5.4 2024-03-26T21:30:37.939 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:30:38.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:38.023 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:38.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:38.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:38.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:38.376 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:38.400 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:38.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:38.544 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: pgmap v56: 132 pgs: 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 84/594 objects degraded (14.141%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 21 pgs peering) 2024-03-26T21:30:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T21:30:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: osdmap e95: 8 total, 7 up, 8 in 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:30:38.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:30:38.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: pgmap v56: 132 pgs: 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:38.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 84/594 objects degraded (14.141%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:38.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 21 pgs peering) 2024-03-26T21:30:38.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T21:30:38.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: osdmap e95: 8 total, 7 up, 8 in 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:30:38.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:30:38.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:38.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:38.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.791 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:38.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:38.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:38.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:38.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:38.961 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: pgmap v56: 132 pgs: 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 84/594 objects degraded (14.141%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 21 pgs peering) 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1412767684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: osdmap e95: 8 total, 7 up, 8 in 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:30:38.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:30:38.963 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:30:38.963 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:30:38.963 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:38 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:30:39.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:39.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:39.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:39.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:39.304 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:39.314 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:30:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:39.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:39.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:39.438 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:30:39.457 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:30:39.476 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:30:39.495 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:30:39.515 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:30:39.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:39.539 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:30:39.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:39.561 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:30:39.561 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:30:39.561 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:30:39.561 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:30:39.562 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:30:39.562 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:30:39.562 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:30:39.562 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:30:39.562 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: osdmap e96: 8 total, 7 up, 8 in 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:30:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2683229721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: osdmap e96: 8 total, 7 up, 8 in 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:30:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:39 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2683229721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:39.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:39.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:39.780 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: osdmap e96: 8 total, 7 up, 8 in 2024-03-26T21:30:39.780 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:30:39.781 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:39 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2683229721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:39.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.811 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:39.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:39.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:39.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:40.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:40.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:40.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:40.188 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:40.199 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T21:30:40.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.316 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:40.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:40.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:30:40.532 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:30:40.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:40.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:40.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:40.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:40 smithi067 ceph-mon[33346]: pgmap v59: 148 pgs: 16 unknown, 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:40 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:40 smithi067 ceph-mon[33346]: osdmap e97: 8 total, 7 up, 8 in 2024-03-26T21:30:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:40 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:30:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:40 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:30:40.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:40.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[40587]: pgmap v59: 148 pgs: 16 unknown, 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:40.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:40.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[40587]: osdmap e97: 8 total, 7 up, 8 in 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[29406]: pgmap v59: 148 pgs: 16 unknown, 22 active+undersized+degraded, 36 active+undersized, 74 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[29406]: osdmap e97: 8 total, 7 up, 8 in 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:30:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:40 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:30:40.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:40.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:40.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:40.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:41.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:41.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:41.175 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:41 smithi067 systemd[1]: Stopping Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:41.189 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:41.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:41.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.347 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:41.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:41.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.472 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:41.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:41.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.598 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:41.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:41.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:41.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:41.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.850 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:41.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:41.905 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:30:41.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:41.975 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:41.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:41.992 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:30:41.992 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:30:42.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:42.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:42.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:42.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:42.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:42.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:42.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-26T21:30:42.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:42.533 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:30:42.550 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:30:42.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:42.607 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:42.620 INFO:teuthology.orchestra.run.smithi067.stderr:Exporting 5.5 info 5.5( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-03-26T21:30:42.620 INFO:teuthology.orchestra.run.smithi067.stderr:Export successful 2024-03-26T21:30:42.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:42.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: pgmap v61: 148 pgs: 11 unknown, 22 active+undersized+degraded, 37 active+undersized, 78 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: osdmap e98: 8 total, 7 up, 8 in 2024-03-26T21:30:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:30:42.739 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.475Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T21:30:42.741 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T21:30:42.742 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T21:30:42.742 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T21:30:42.742 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[62298]: ts=2024-03-26T21:30:42.476Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T21:30:42.742 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 podman[66991]: 2024-03-26 21:30:42.486133396 +0000 UTC m=+0.283579082 container died 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: pgmap v61: 148 pgs: 11 unknown, 22 active+undersized+degraded, 37 active+undersized, 78 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: osdmap e98: 8 total, 7 up, 8 in 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:30:42.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: pgmap v61: 148 pgs: 11 unknown, 22 active+undersized+degraded, 37 active+undersized, 78 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: osdmap e98: 8 total, 7 up, 8 in 2024-03-26T21:30:42.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:42 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:30:42.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.860 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:42.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:42.961 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:42.985 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:42.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:43.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:30:43.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:43.117 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:43.128 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 podman[66991]: 2024-03-26 21:30:42.884770083 +0000 UTC m=+0.682215762 container cleanup 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:43.128 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:42 smithi067 bash[66991]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T21:30:43.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:43.242 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:43.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:43.369 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:43.456 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 podman[67211]: 2024-03-26 21:30:43.136009302 +0000 UTC m=+0.652320430 container remove 22f58cc11b1f4d0fc29f1f94ce1c6c8726e924eaec15cb574187fc78be4bbd2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:43.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:43.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:43.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[40587]: osdmap e99: 8 total, 7 up, 8 in 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:30:43.706 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:43 smithi055 ceph-mon[29406]: osdmap e99: 8 total, 7 up, 8 in 2024-03-26T21:30:43.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:43 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1057470561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:30:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:43 smithi067 ceph-mon[33346]: osdmap e99: 8 total, 7 up, 8 in 2024-03-26T21:30:43.740 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T21:30:43.740 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 systemd[1]: Stopped Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:43.740 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.140s CPU time. 2024-03-26T21:30:43.740 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 systemd[1]: Starting Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:30:43.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:43.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.4 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 5.5 2024-03-26T21:30:43.800 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:43.801 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:43.816 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:43.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.873 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:43.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:43.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:43.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:43.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T21:30:44.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:44.109 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:43 smithi067 podman[67617]: 2024-03-26 21:30:43.876762787 +0000 UTC m=+0.022351909 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T21:30:44.109 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 podman[67617]: 2024-03-26 21:30:44.016224541 +0000 UTC m=+0.161813648 container create ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:44.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:44.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:44.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:44.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:44.379 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:44.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 podman[67617]: 2024-03-26 21:30:44.26093535 +0000 UTC m=+0.406524458 container init ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 podman[67617]: 2024-03-26 21:30:44.263637372 +0000 UTC m=+0.409226480 container start ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.287Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.287Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.287Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi067 (none))" 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.287Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.287Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.289Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.289Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T21:30:44.490 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.291Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T21:30:44.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.291Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T21:30:44.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.292Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T21:30:44.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.292Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.676µs 2024-03-26T21:30:44.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.292Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.293Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.294Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.294Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.878µs wal_replay_duration=1.681933ms wbl_replay_duration=126ns total_replay_duration=1.727874ms 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.295Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.295Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.295Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.318Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.836965ms db_storage=742ns remote_storage=1.055µs web_handler=311ns query_engine=469ns scrape=537.974µs scrape_sd=93.167µs notify=10.504µs notify_sd=8.553µs rules=21.897777ms tracing=5.93µs 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.318Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:30:44.318Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 bash[67617]: ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d 2024-03-26T21:30:44.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:30:44 smithi067 systemd[1]: Started Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:30:44.505 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:44.550 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPING 2024-03-26T21:30:44.551 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:30:44.551 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPED 2024-03-26T21:30:44.551 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTING 2024-03-26T21:30:44.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:44.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:44.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:44.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: pgmap v64: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:44.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: pgmap v64: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Serving on http://:::9283 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTED 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPING 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPED 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTING 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Serving on http://:::9283 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTED 2024-03-26T21:30:44.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPING 2024-03-26T21:30:44.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:30:44.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STOPPED 2024-03-26T21:30:44.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTING 2024-03-26T21:30:44.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Serving on http://:::9283 2024-03-26T21:30:44.819 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:44 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:30:44] ENGINE Bus STARTED 2024-03-26T21:30:44.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:44.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:44.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:44.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: pgmap v64: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:44.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:44.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:44.993 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:44.993 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.994 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:44.994 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:44.994 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:44.994 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:30:45.006 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:45.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:45.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:45.257 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:45.359 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:45.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:30:45.508 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:45.559 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:30:45.383Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003349828s 2024-03-26T21:30:45.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:45.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:30:45.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:45 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:45.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:45.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:45.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:45.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:45 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:30:45.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:45.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:45.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:46.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:46.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:46.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:46.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:46.227 INFO:teuthology.orchestra.run.smithi067.stdout: marking collection for removal 2024-03-26T21:30:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:setting '_remove' omap key 2024-03-26T21:30:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:finish_remove_pgs 5.5_head removing 5.5 2024-03-26T21:30:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:Remove successful 2024-03-26T21:30:46.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:46.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:46.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.431 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:46.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:46.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:46.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.556 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:46.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:46.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:46 smithi055 ceph-mon[40587]: pgmap v65: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:46.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:46 smithi055 ceph-mon[29406]: pgmap v65: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:46.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:46.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:46.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.808 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:46.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:46.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:46.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:46.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:46 smithi067 ceph-mon[33346]: pgmap v65: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:47.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:47.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:47.161 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.185 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:47.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:47.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.4 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.5.4 2024-03-26T21:30:47.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.4/config 2024-03-26T21:30:47.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:47.563 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:47.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.690 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:47.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:47.798 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:47.799 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:47.815 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:47.827 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:47.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:47.944 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:47.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:48.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.075 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:48.093 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:30:48.093 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:30:48.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.201 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:48.302 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:48.326 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:48.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:48.450 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:48.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:48.577 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:48.671 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:48 smithi055 ceph-mon[40587]: pgmap v66: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:48.671 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:48 smithi055 ceph-mon[29406]: pgmap v66: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:48.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:48.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:30:48.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:48.828 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:48 smithi067 ceph-mon[33346]: pgmap v66: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:48.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:48.952 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:48.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:49.031 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:30:49.041 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:30:49.054 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:49.061 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:30:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:49.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:30:49.102 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:30:49.122 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:30:49.143 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:30:49.143 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:30:49.144 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:30:49.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:49.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:49.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:49.392 INFO:teuthology.orchestra.run.smithi067.stdout:get_pg_num_history pg_num_history pg_num_history(e92 pg_nums {1={19=1},2={66=3},3={66=32},4={68=32},5={70=32},6={72=32}} deleted_pools ) 2024-03-26T21:30:49.393 INFO:teuthology.orchestra.run.smithi067.stdout:Importing pgid 5.5 2024-03-26T21:30:49.393 INFO:teuthology.orchestra.run.smithi067.stdout:write_pg epoch 91 info 5.5( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-03-26T21:30:49.393 INFO:teuthology.orchestra.run.smithi067.stdout:Import successful 2024-03-26T21:30:49.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:49.455 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:49.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:49.585 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:49.592 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-26T21:30:49.646 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3565438331' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.646 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:49 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.646 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1893563256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:49.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:49.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:49.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3565438331' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1893563256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3565438331' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:30:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1893563256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.839 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:49.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:49.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:49.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:49.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:30:50.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:30:50.016 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:50.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.093 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:50.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:50.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.219 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:50.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:50.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.345 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:50.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:50.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:50.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/exp.5.5.4 2024-03-26T21:30:50.470 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:50.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.596 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:50.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:50.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.722 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:50.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:50.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:50 smithi067 ceph-mon[33346]: pgmap v67: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:50.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:50 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:50.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:50 smithi067 ceph-mon[33346]: osdmap e100: 8 total, 7 up, 8 in 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[40587]: pgmap v67: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[40587]: osdmap e100: 8 total, 7 up, 8 in 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[29406]: pgmap v67: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:30:50.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:50 smithi055 ceph-mon[29406]: osdmap e100: 8 total, 7 up, 8 in 2024-03-26T21:30:50.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:30:50.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:50.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:50.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:50.975 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:50.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:51.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:51.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:51.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:51.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:51.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:51.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:51.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:51.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:51.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.734 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:51.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:51 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:30:51.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:51.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:51.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:51.984 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:51.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:52.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:30:52.041 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:51 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:30:52.041 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:51 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:30:52.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:52.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:52.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.234 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:52.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:52.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:52.358 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:52.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:52.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:52.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:52.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:52.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.736 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:52.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:52.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[40587]: pgmap v69: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 586 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:52.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:52.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:52.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[29406]: pgmap v69: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 586 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:52.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:52.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:52 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:52.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:52.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:52.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:52.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:52.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:52 smithi067 ceph-mon[33346]: pgmap v69: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 586 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:52.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:52 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:52.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:52 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:30:53.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:53.109 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:53.114 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:53.120 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:30:53.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:53.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:53.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.366 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:53.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:53.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:53.491 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:53.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:53.598 DEBUG:teuthology.orchestra.run.smithi055:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-26T21:30:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:53.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:53.619 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T21:30:53.619 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T21:30:53.696 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:hints = 1 2024-03-26T21:30:53.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T21:30:53.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object prefix: benchmark_data_smithi055_81491 2024-03-26T21:30:53.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:30:53.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 0 0 0 0 0 0 - 0 2024-03-26T21:30:53.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:53.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:53.880 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:53.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:53.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:54.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.163 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:54.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:54.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:54.303 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:54.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:54.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:54.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:54.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:54.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 1 16 3264 3248 202.97 203 0.00243389 0.00490344 2024-03-26T21:30:54.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:54.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:54.874 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:54.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:54.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:54 smithi067 ceph-mon[33346]: pgmap v70: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:55.017 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:54 smithi055 ceph-mon[40587]: pgmap v70: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:54 smithi055 ceph-mon[29406]: pgmap v70: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:55.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:55.160 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:55.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:55.304 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:55.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:55.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:55.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T21:30:55.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:30:55.499 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-26T21:30:55.499 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:30:55.499 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:30:55.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-26T21:30:55.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:55.584 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:55.657 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:30:55.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:55.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 2 15 6292 6277 196.138 189.312 0.00646365 0.00508336 2024-03-26T21:30:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.742 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:30:55.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:55.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:55.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:55.950 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1638919214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:55.950 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:55.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:56.027 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:56.034 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1638919214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:56.034 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:56.035 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1638919214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:56.035 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:30:56.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:56.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:56.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:56.402 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:56.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:56.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:56.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:56.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:30:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:30:56.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:56.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 3 16 9043 9027 188.045 171.875 0.00274006 0.005244 2024-03-26T21:30:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.702 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:56.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:56.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:56.833 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:56.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:30:56.976 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:56.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:30:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:56 smithi067 ceph-mon[33346]: pgmap v71: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:56 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:56 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:30:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:56 smithi067 ceph-mon[33346]: osdmap e101: 8 total, 7 up, 8 in 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[40587]: pgmap v71: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[40587]: osdmap e101: 8 total, 7 up, 8 in 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[29406]: pgmap v71: 148 pgs: 22 active+undersized+degraded, 41 active+undersized, 85 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/594 objects degraded (14.141%) 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:30:57.042 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:56 smithi055 ceph-mon[29406]: osdmap e101: 8 total, 7 up, 8 in 2024-03-26T21:30:57.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:57.120 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:57.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:57.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:57.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:57.384 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:57.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:57.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:57.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:57.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:57.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 4 16 11759 11743 183.468 169.75 0.0216184 0.00543027 2024-03-26T21:30:57.708 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:30:57.708 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:30:57.708 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 0 2024-03-26T21:30:57.708 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:30:57.739 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:30:57.740 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T21:30:57.740 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:30:57.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:57.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:57.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:57.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:57.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:30:57.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:57.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:30:57.936 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:57 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:30:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:57.936 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:57.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:30:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:57 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:30:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:57 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:30:58.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:58.067 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:58.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:30:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:30:58.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:58.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:58.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:58.493 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:30:58.537 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:30:58.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:58.582 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:30:58.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:58.626 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:30:58.669 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:30:58.697 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 5 16 14366 14350 179.359 162.938 0.0110189 0.00556628 2024-03-26T21:30:58.710 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:30:58.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:30:58.752 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:30:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:30:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:30:58.752 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:30:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:30:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:30:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:30:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:30:58.753 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:30:58.766 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:30:58.790 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[40587]: pgmap v73: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:30:58.790 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2646/24786 objects degraded (10.675%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:58.790 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[40587]: osdmap e102: 8 total, 7 up, 8 in 2024-03-26T21:30:58.790 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2123748777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:58.791 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[29406]: pgmap v73: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:30:58.791 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2646/24786 objects degraded (10.675%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:58.791 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[29406]: osdmap e102: 8 total, 7 up, 8 in 2024-03-26T21:30:58.791 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2123748777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:58.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:58.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:58.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:58 smithi067 ceph-mon[33346]: pgmap v73: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:30:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:58 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2646/24786 objects degraded (10.675%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:30:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:58 smithi067 ceph-mon[33346]: osdmap e102: 8 total, 7 up, 8 in 2024-03-26T21:30:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:30:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2123748777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:30:59.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:59.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:59.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:30:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:30:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:30:59.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:30:59.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:30:59.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:30:59.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:30:59.364 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:30:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:30:59.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:30:59.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:30:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:30:59.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:30:59.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:30:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:59.699 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 6 16 16747 16731 174.201 148.812 0.00566194 0.00573232 2024-03-26T21:30:59.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:59.753 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:59.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:30:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:30:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:30:59.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:30:59.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:00.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:00.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:00.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:00.276 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:00.328 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:00.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:00.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:00.408 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:00.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:00.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:00.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:00.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 7 16 19219 19203 171.386 154.5 0.00386079 0.00582838 2024-03-26T21:31:00.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[29406]: pgmap v75: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[40587]: pgmap v75: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:00.779 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:00.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:00.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:00.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:00.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:00.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:00 smithi067 ceph-mon[33346]: pgmap v75: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 2646/24786 objects degraded (10.675%) 2024-03-26T21:31:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:00 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:01.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:01.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:01.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.254 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:01.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:01.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:01.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:01.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:01.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:01.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:31:01.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:01.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:01.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 8 16 21500 21484 167.782 142.562 0.00398185 0.00595479 2024-03-26T21:31:01.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:01.807 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:01.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:01.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:01.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:02.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.081 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:02.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:02.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:02.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.365 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:02.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:02.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:02.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:02.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 9 16 23648 23632 164.056 134.25 0.00515519 0.00608919 2024-03-26T21:31:02.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:02.774 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:02.875 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:02.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:02.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:02 smithi067 ceph-mon[33346]: pgmap v76: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 753 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s; 3464/36723 objects degraded (9.433%) 2024-03-26T21:31:03.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:03.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:02 smithi055 ceph-mon[40587]: pgmap v76: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 753 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s; 3464/36723 objects degraded (9.433%) 2024-03-26T21:31:03.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:02 smithi055 ceph-mon[29406]: pgmap v76: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 753 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s; 3464/36723 objects degraded (9.433%) 2024-03-26T21:31:03.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:03.275 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:03.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:03.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:03.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:03.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:03.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:03.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 10 16 25351 25335 158.294 106.438 0.00538145 0.0062841 2024-03-26T21:31:03.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:03.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:03.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:03.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T21:31:03.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:31:03.935 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T21:31:03.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:31:03.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:03.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:04.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:04.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:03 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6601/62532 objects degraded (10.556%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:04.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:03 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6601/62532 objects degraded (10.556%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:04.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:04.209 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:03 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6601/62532 objects degraded (10.556%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:04.311 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:04.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:04.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:04.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:04.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:04.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:04.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 11 16 26771 26755 151.972 88.75 0.00261857 0.00651816 2024-03-26T21:31:04.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:04.763 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:04.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:04.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:04.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:05.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[40587]: pgmap v77: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.58k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[40587]: osdmap e103: 8 total, 7 up, 8 in 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[29406]: pgmap v77: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.58k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:31:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:04 smithi055 ceph-mon[29406]: osdmap e103: 8 total, 7 up, 8 in 2024-03-26T21:31:05.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:05.168 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:04 smithi067 ceph-mon[33346]: pgmap v77: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.58k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:04 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:31:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:04 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:31:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:04 smithi067 ceph-mon[33346]: osdmap e103: 8 total, 7 up, 8 in 2024-03-26T21:31:05.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.296 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:05.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:05.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:05.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:05.565 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:05.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.691 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:05.702 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 12 16 27956 27940 145.48 74.0625 0.00218988 0.00680766 2024-03-26T21:31:05.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:05.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:05.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:05.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:05.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:06.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:06.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:06.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:06.228 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:06.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:06.353 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:06.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.496 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:06.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:06.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:06.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:06.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 13 16 29323 29307 140.862 85.4375 0.00420937 0.00709346 2024-03-26T21:31:06.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:06.752 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:06.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:06.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:06.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:06.985 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.015 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:07.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:07.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:06 smithi055 ceph-mon[40587]: pgmap v79: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.57k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:06 smithi055 ceph-mon[29406]: pgmap v79: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.57k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:07.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:06 smithi067 ceph-mon[33346]: pgmap v79: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.57k op/s; 6601/62532 objects degraded (10.556%) 2024-03-26T21:31:07.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.286 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:07.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:07.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:07.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:07.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:07.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:07.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:07.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:07.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 14 16 31007 30991 138.317 105.25 0.00146613 0.00720874 2024-03-26T21:31:07.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:07.822 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:07.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:07.952 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:07.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:08.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:08.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:08.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.216 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:08.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:08.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:08.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:08.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:08.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:08.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:08.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:08.700 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 15 16 32455 32439 135.129 90.5 0.00960898 0.00739413 2024-03-26T21:31:08.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:08.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:08.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:08.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:08.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:08.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:09.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:08 smithi055 ceph-mon[29406]: pgmap v80: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:09.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:08 smithi055 ceph-mon[40587]: pgmap v80: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:09.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:09.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:09.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:08 smithi067 ceph-mon[33346]: pgmap v80: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:09.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:09.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:09.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:09.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:09.539 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:09.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:09.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 16 16 33960 33944 132.563 94.0625 0.0107487 0.00753757 2024-03-26T21:31:09.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:09.815 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:09.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:09.959 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:09.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:10.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:10.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:10.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:10.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:10.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:10.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:10.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:10.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:10.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:10.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 17 15 35588 35573 130.752 101.812 0.00830442 0.00764078 2024-03-26T21:31:10.766 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:10.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:10.906 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:10.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:11.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:11.035 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:10 smithi055 ceph-mon[29406]: pgmap v81: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:10 smithi055 ceph-mon[40587]: pgmap v81: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:11.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:10 smithi067 ceph-mon[33346]: pgmap v81: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.8 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 9210/87285 objects degraded (10.552%) 2024-03-26T21:31:11.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:11.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:11.318 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:11.319 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:11.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:11.418 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:31:11.418 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:31:11.467 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:11.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:11.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:11.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 18 16 37065 37049 128.613 92.25 0.00310983 0.00776448 2024-03-26T21:31:11.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:11.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:11.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:11.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:12.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:12.059 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:12.161 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:12.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[40587]: pgmap v82: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.9 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 9210/91503 objects degraded (10.065%) 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 9210/87285 objects degraded (10.552%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[29406]: pgmap v82: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.9 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 9210/91503 objects degraded (10.065%) 2024-03-26T21:31:12.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:12 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 9210/87285 objects degraded (10.552%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:12.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:12.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:12.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:12.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:12.489 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:12 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:12 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:12 smithi067 ceph-mon[33346]: pgmap v82: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 1.9 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s; 9210/91503 objects degraded (10.065%) 2024-03-26T21:31:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:12 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 9210/87285 objects degraded (10.552%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:12.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:12.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:12.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:12.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:31:12.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 19 16 38769 38753 127.446 106.5 0.00865398 0.00784004 2024-03-26T21:31:12.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:12.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:12.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:12.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:12.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:13.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:13.077 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:13.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:13.220 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:13.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:13.400 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:13.401 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:13.401 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:13.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:13.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:13.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:13 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:13.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:13 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:31:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:13 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:31:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:13 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:13.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:13.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.637 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:13.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:13.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:31:13.709803+0000 min lat: 0.00119611 max lat: 0.190367 avg lat: 0.00792136 2024-03-26T21:31:13.702 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:31:13.702 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 20 16 40322 40306 125.927 97.0625 0.00193529 0.00792136 2024-03-26T21:31:13.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:13.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:13.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:13.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:14.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:14.038 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:14.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:14.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[40587]: pgmap v83: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:14.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/721123525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.173 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:31:14.173 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[29406]: pgmap v83: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:14.173 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/721123525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.173 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:14 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.185 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:14.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:14.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:14.324 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:14.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:14.455 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:14 smithi067 ceph-mon[33346]: pgmap v83: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/721123525' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:14 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:31:14.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:14.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:14.701 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 21 16 42023 42007 124.992 106.312 0.00873505 0.0079954 2024-03-26T21:31:14.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:14.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:14.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:14.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:14.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:14.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:15.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:15.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:15.163 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:15.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:15.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:15.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:15.481 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:15.481 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[29406]: osdmap e104: 8 total, 7 up, 8 in 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[40587]: osdmap e104: 8 total, 7 up, 8 in 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:31:15.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:15 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:31:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:15 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:15 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:31:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:15 smithi067 ceph-mon[33346]: osdmap e104: 8 total, 7 up, 8 in 2024-03-26T21:31:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:15 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:31:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:15 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:31:15.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:15.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:15.702 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 22 16 43745 43729 124.202 107.625 0.00632546 0.00804627 2024-03-26T21:31:15.712 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:15.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:15.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:15.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:15.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:15.983 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:16.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.129 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:16.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:16.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:16.402 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:31:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:31:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:31:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:31:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:31:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:31:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: osdmap e105: 8 total, 7 up, 8 in 2024-03-26T21:31:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: pgmap v86: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:16.505 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:16.543 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:16.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: osdmap e105: 8 total, 7 up, 8 in 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: pgmap v86: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:31:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: osdmap e105: 8 total, 7 up, 8 in 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: pgmap v86: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s; 11654/110124 objects degraded (10.583%) 2024-03-26T21:31:16.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:16.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:16.703 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 23 15 45462 45447 123.47 107.375 0.0153261 0.00809388 2024-03-26T21:31:16.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.822 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:16.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:16.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:16.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:16.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:16.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:17.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:17.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:17.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:17.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:17.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:31:17.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:17.372 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:17.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:17 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:31:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:17 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:31:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:17 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 11654/110124 objects degraded (10.583%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:17 smithi067 ceph-mon[33346]: osdmap e106: 8 total, 7 up, 8 in 2024-03-26T21:31:17.502 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 11654/110124 objects degraded (10.583%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[29406]: osdmap e106: 8 total, 7 up, 8 in 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 11654/110124 objects degraded (10.583%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:17.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:17 smithi055 ceph-mon[40587]: osdmap e106: 8 total, 7 up, 8 in 2024-03-26T21:31:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.522 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:17.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:17.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:17.663 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:31:17.705 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 24 16 46974 46958 122.256 94.4375 0.00645256 0.00817492 2024-03-26T21:31:17.708 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:31:17.750 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:31:17.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.779 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:17.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:17.792 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:31:17.847 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:31:17.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:17.891 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:31:17.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:17.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:31:17.932 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:31:18.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:18.084 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:18.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:18.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:18.372 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:18.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: pgmap v88: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3271207837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.523 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: pgmap v88: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3271207837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: pgmap v88: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:18.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3271207837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:18.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:18.654 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:18.703 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 25 15 48452 48437 121.062 92.4375 0.00681565 0.00825443 2024-03-26T21:31:18.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.797 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:18.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:18.938 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:18.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:19.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:19.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:19.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:19.234 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:31:19.244 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[40587]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:31:19.245 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:31:19.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:31:19.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:31:19.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:31:19.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:19 smithi055 ceph-mon[29406]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:31:19.246 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:19.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.368 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:19.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:19.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:19 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:31:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:19 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:31:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:19 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:31:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:19 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:31:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:19 smithi067 ceph-mon[33346]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:31:19.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:19.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.640 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:19.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:19.703 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 26 16 50033 50017 120.204 98.75 0.00500725 0.00831395 2024-03-26T21:31:19.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:19.770 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:19.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:19.898 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:19.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:20.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:20.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:20.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:20.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:20.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:20.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:20.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:20.432 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:20 smithi067 ceph-mon[33346]: pgmap v89: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:20.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:20.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:20.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:20 smithi055 ceph-mon[40587]: pgmap v89: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:20.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:20 smithi055 ceph-mon[29406]: pgmap v89: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s; 14148/134973 objects degraded (10.482%) 2024-03-26T21:31:20.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:20.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:20.704 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 27 16 51451 51435 119.034 88.625 0.0123297 0.00839495 2024-03-26T21:31:20.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:20.835 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:20.937 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:20.975 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:20.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:21.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:21.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:21.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:21.246 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:21.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:21.477 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:21 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:31:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:21 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:31:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:21.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:21.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:21 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:31:21.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:21 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:31:21.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:21 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:31:21.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:21 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:31:21.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:21.703 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 28 16 52899 52883 118.015 90.5 0.00672044 0.00846857 2024-03-26T21:31:21.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.793 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:21.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:31:21.841 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:31:21.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:21.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:21.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:22.038 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:22.093 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:22.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:22.240 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:22.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:22.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:22.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:22.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[40587]: pgmap v90: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 717 B/s rd, 85 MiB/s wr, 1.36k op/s; 14148/139218 objects degraded (10.162%) 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 14148/134973 objects degraded (10.482%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[40587]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[29406]: pgmap v90: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 717 B/s rd, 85 MiB/s wr, 1.36k op/s; 14148/139218 objects degraded (10.162%) 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 14148/134973 objects degraded (10.482%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[29406]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T21:31:22.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:31:22.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:22 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:31:22.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:22.665 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:22.701 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:22.702 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-03-26T21:31:17.303405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2936150440,"num_objects":44991,"num_object_clones":0,"num_object_copies":134973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44991,"num_whiteouts":0,"num_read":1081,"num_read_kb":927,"num_write":45408,"num_write_kb":2869213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45424,"ondisk_log_size":45424,"up":381,"acting":381,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":439,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7920528,"kb_used_data":7564224,"kb_used_omap":12,"kb_used_meta":356211,"kb_avail":742040688,"statfs":{"total":767960285184,"available":759849664512,"internally_reserved":0,"allocated":7745765376,"data_stored":7742587598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":364760624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":949616640,"num_objects":14490,"num_object_clones":0,"num_object_copies":43470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14490,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14490,"num_write_kb":927360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003136"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.170054+0000","last_change":"2024-03-26T21:30:31.738275+0000","last_active":"2024-03-26T21:31:15.170054+0000","last_peered":"2024-03-26T21:31:15.170054+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738275+0000","last_became_peered":"2024-03-26T21:30:31.738275+0000","last_unstale":"2024-03-26T21:31:15.170054+0000","last_undegraded":"2024-03-26T21:31:15.170054+0000","last_fullsized":"2024-03-26T21:30:31.732719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:32.238385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.174208+0000","last_change":"2024-03-26T21:30:12.067262+0000","last_active":"2024-03-26T21:31:15.174208+0000","last_peered":"2024-03-26T21:31:15.174208+0000","last_clean":"2024-03-26T21:31:15.174208+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:31:15.174208+0000","last_undegraded":"2024-03-26T21:31:15.174208+0000","last_fullsized":"2024-03-26T21:31:15.174208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.067176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.067176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:37.851966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184010+0000","last_change":"2024-03-26T21:30:31.663369+0000","last_active":"2024-03-26T21:31:15.184010+0000","last_peered":"2024-03-26T21:31:15.184010+0000","last_clean":"2024-03-26T21:31:15.184010+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:31:15.184010+0000","last_undegraded":"2024-03-26T21:31:15.184010+0000","last_fullsized":"2024-03-26T21:31:15.184010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:30:31.663239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:44.743894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":66,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.179167+0000","last_change":"2024-03-26T21:30:31.740069+0000","last_active":"2024-03-26T21:31:15.179167+0000","last_peered":"2024-03-26T21:31:15.179167+0000","last_clean":"2024-03-26T21:28:42.030750+0000","last_became_active":"2024-03-26T21:30:31.740069+0000","last_became_peered":"2024-03-26T21:30:31.740069+0000","last_unstale":"2024-03-26T21:31:15.179167+0000","last_undegraded":"2024-03-26T21:30:31.731923+0000","last_fullsized":"2024-03-26T21:30:31.731769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179615+0000","last_change":"2024-03-26T21:29:49.749475+0000","last_active":"2024-03-26T21:31:15.179615+0000","last_peered":"2024-03-26T21:31:15.179615+0000","last_clean":"2024-03-26T21:31:15.179615+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:31:15.179615+0000","last_undegraded":"2024-03-26T21:31:15.179615+0000","last_fullsized":"2024-03-26T21:31:15.179615+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:49.749386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:49.749386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.178903+0000","last_change":"2024-03-26T21:30:31.739659+0000","last_active":"2024-03-26T21:31:15.178903+0000","last_peered":"2024-03-26T21:31:15.178903+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.739659+0000","last_became_peered":"2024-03-26T21:30:31.739659+0000","last_unstale":"2024-03-26T21:31:15.178903+0000","last_undegraded":"2024-03-26T21:31:15.178903+0000","last_fullsized":"2024-03-26T21:30:31.731768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180256+0000","last_change":"2024-03-26T21:29:50.735279+0000","last_active":"2024-03-26T21:31:15.180256+0000","last_peered":"2024-03-26T21:31:15.180256+0000","last_clean":"2024-03-26T21:31:15.180256+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:31:15.180256+0000","last_undegraded":"2024-03-26T21:31:15.180256+0000","last_fullsized":"2024-03-26T21:31:15.180256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.735199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.735199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":117,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179787+0000","last_change":"2024-03-26T21:30:05.728263+0000","last_active":"2024-03-26T21:31:15.179787+0000","last_peered":"2024-03-26T21:31:15.179787+0000","last_clean":"2024-03-26T21:31:15.179787+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:31:15.179787+0000","last_undegraded":"2024-03-26T21:31:15.179787+0000","last_fullsized":"2024-03-26T21:31:15.179787+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:05.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.728097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.255731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183242+0000","last_change":"2024-03-26T21:29:58.804717+0000","last_active":"2024-03-26T21:31:15.183242+0000","last_peered":"2024-03-26T21:31:15.183242+0000","last_clean":"2024-03-26T21:31:15.183242+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:31:15.183242+0000","last_undegraded":"2024-03-26T21:31:15.183242+0000","last_fullsized":"2024-03-26T21:31:15.183242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.804636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.804636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.335495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169658+0000","last_change":"2024-03-26T21:30:23.539609+0000","last_active":"2024-03-26T21:31:15.169658+0000","last_peered":"2024-03-26T21:31:15.169658+0000","last_clean":"2024-03-26T21:31:15.169658+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:31:15.169658+0000","last_undegraded":"2024-03-26T21:31:15.169658+0000","last_fullsized":"2024-03-26T21:31:15.169658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:30:23.539529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:38.882369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.169850+0000","last_change":"2024-03-26T21:30:31.738978+0000","last_active":"2024-03-26T21:31:15.169850+0000","last_peered":"2024-03-26T21:31:15.169850+0000","last_clean":"2024-03-26T21:30:22.526419+0000","last_became_active":"2024-03-26T21:30:31.738978+0000","last_became_peered":"2024-03-26T21:30:31.738978+0000","last_unstale":"2024-03-26T21:31:15.169850+0000","last_undegraded":"2024-03-26T21:31:15.169850+0000","last_fullsized":"2024-03-26T21:30:31.733643+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:22.526329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:22.526329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:24.965973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":66,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.181304+0000","last_change":"2024-03-26T21:30:31.742645+0000","last_active":"2024-03-26T21:31:15.181304+0000","last_peered":"2024-03-26T21:31:15.181304+0000","last_clean":"2024-03-26T21:28:42.145195+0000","last_became_active":"2024-03-26T21:30:31.742645+0000","last_became_peered":"2024-03-26T21:30:31.742645+0000","last_unstale":"2024-03-26T21:31:15.181304+0000","last_undegraded":"2024-03-26T21:30:31.730765+0000","last_fullsized":"2024-03-26T21:30:31.730641+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169288+0000","last_change":"2024-03-26T21:30:11.468492+0000","last_active":"2024-03-26T21:31:15.169288+0000","last_peered":"2024-03-26T21:31:15.169288+0000","last_clean":"2024-03-26T21:31:15.169288+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:31:15.169288+0000","last_undegraded":"2024-03-26T21:31:15.169288+0000","last_fullsized":"2024-03-26T21:31:15.169288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.468385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.517400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177038+0000","last_change":"2024-03-26T21:30:31.744378+0000","last_active":"2024-03-26T21:31:15.177038+0000","last_peered":"2024-03-26T21:31:15.177038+0000","last_clean":"2024-03-26T21:30:19.144400+0000","last_became_active":"2024-03-26T21:30:31.744378+0000","last_became_peered":"2024-03-26T21:30:31.744378+0000","last_unstale":"2024-03-26T21:31:15.177038+0000","last_undegraded":"2024-03-26T21:31:15.177038+0000","last_fullsized":"2024-03-26T21:30:31.735352+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:07.539633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.539633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.037600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.169625+0000","last_change":"2024-03-26T21:30:31.738170+0000","last_active":"2024-03-26T21:31:15.169625+0000","last_peered":"2024-03-26T21:31:15.169625+0000","last_clean":"2024-03-26T21:30:28.546608+0000","last_became_active":"2024-03-26T21:30:31.738170+0000","last_became_peered":"2024-03-26T21:30:31.738170+0000","last_unstale":"2024-03-26T21:31:15.169625+0000","last_undegraded":"2024-03-26T21:31:15.169625+0000","last_fullsized":"2024-03-26T21:30:31.731908+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:28.546501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:28.546501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:52.052216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":60,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.179128+0000","last_change":"2024-03-26T21:30:31.745243+0000","last_active":"2024-03-26T21:31:15.179128+0000","last_peered":"2024-03-26T21:31:15.179128+0000","last_clean":"2024-03-26T21:28:42.123479+0000","last_became_active":"2024-03-26T21:30:31.745243+0000","last_became_peered":"2024-03-26T21:30:31.745243+0000","last_unstale":"2024-03-26T21:31:15.179128+0000","last_undegraded":"2024-03-26T21:30:31.731435+0000","last_fullsized":"2024-03-26T21:30:31.731307+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179866+0000","last_change":"2024-03-26T21:30:31.740216+0000","last_active":"2024-03-26T21:31:15.179866+0000","last_peered":"2024-03-26T21:31:15.179866+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.740216+0000","last_became_peered":"2024-03-26T21:30:31.740216+0000","last_unstale":"2024-03-26T21:31:15.179866+0000","last_undegraded":"2024-03-26T21:31:15.179866+0000","last_fullsized":"2024-03-26T21:30:31.732999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.175648+0000","last_change":"2024-03-26T21:30:31.748769+0000","last_active":"2024-03-26T21:31:15.175648+0000","last_peered":"2024-03-26T21:31:15.175648+0000","last_clean":"2024-03-26T21:30:19.057549+0000","last_became_active":"2024-03-26T21:30:31.748769+0000","last_became_peered":"2024-03-26T21:30:31.748769+0000","last_unstale":"2024-03-26T21:31:15.175648+0000","last_undegraded":"2024-03-26T21:31:15.175648+0000","last_fullsized":"2024-03-26T21:30:31.734303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.051556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.051556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.009226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.183817+0000","last_change":"2024-03-26T21:30:31.744225+0000","last_active":"2024-03-26T21:31:15.183817+0000","last_peered":"2024-03-26T21:31:15.183817+0000","last_clean":"2024-03-26T21:30:19.056406+0000","last_became_active":"2024-03-26T21:30:31.744225+0000","last_became_peered":"2024-03-26T21:30:31.744225+0000","last_unstale":"2024-03-26T21:31:15.183817+0000","last_undegraded":"2024-03-26T21:31:15.183817+0000","last_fullsized":"2024-03-26T21:30:31.737824+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.708767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.708767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:31.587527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":117,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.172936+0000","last_change":"2024-03-26T21:29:45.654379+0000","last_active":"2024-03-26T21:31:15.172936+0000","last_peered":"2024-03-26T21:31:15.172936+0000","last_clean":"2024-03-26T21:31:15.172936+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:31:15.172936+0000","last_undegraded":"2024-03-26T21:31:15.172936+0000","last_fullsized":"2024-03-26T21:31:15.172936+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:29:45.654238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.654238+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.110313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180882+0000","last_change":"2024-03-26T21:30:32.797018+0000","last_active":"2024-03-26T21:31:15.180882+0000","last_peered":"2024-03-26T21:31:15.180882+0000","last_clean":"2024-03-26T21:31:15.180882+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:31:15.180882+0000","last_undegraded":"2024-03-26T21:31:15.180882+0000","last_fullsized":"2024-03-26T21:31:15.180882+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:32.796920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:32.796920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:58.884872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179570+0000","last_change":"2024-03-26T21:30:31.741526+0000","last_active":"2024-03-26T21:31:15.179570+0000","last_peered":"2024-03-26T21:31:15.179570+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.741526+0000","last_became_peered":"2024-03-26T21:30:31.741526+0000","last_unstale":"2024-03-26T21:31:15.179570+0000","last_undegraded":"2024-03-26T21:31:15.179570+0000","last_fullsized":"2024-03-26T21:30:31.731261+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.173904+0000","last_change":"2024-03-26T21:30:31.740626+0000","last_active":"2024-03-26T21:31:15.173904+0000","last_peered":"2024-03-26T21:31:15.173904+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740626+0000","last_became_peered":"2024-03-26T21:30:31.740626+0000","last_unstale":"2024-03-26T21:31:15.173904+0000","last_undegraded":"2024-03-26T21:31:15.173904+0000","last_fullsized":"2024-03-26T21:30:31.732923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.367500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":110,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.180247+0000","last_change":"2024-03-26T21:30:31.745104+0000","last_active":"2024-03-26T21:31:15.180247+0000","last_peered":"2024-03-26T21:31:15.180247+0000","last_clean":"2024-03-26T21:30:19.791949+0000","last_became_active":"2024-03-26T21:30:31.745104+0000","last_became_peered":"2024-03-26T21:30:31.745104+0000","last_unstale":"2024-03-26T21:31:15.180247+0000","last_undegraded":"2024-03-26T21:30:31.729821+0000","last_fullsized":"2024-03-26T21:30:31.729686+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":93,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177658+0000","last_change":"2024-03-26T21:31:14.781495+0000","last_active":"2024-03-26T21:31:15.177658+0000","last_peered":"2024-03-26T21:31:15.177658+0000","last_clean":"2024-03-26T21:31:15.177658+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:31:15.177658+0000","last_undegraded":"2024-03-26T21:31:15.177658+0000","last_fullsized":"2024-03-26T21:31:15.177658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:14.781351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.781351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.404865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":136,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.180111+0000","last_change":"2024-03-26T21:30:31.738596+0000","last_active":"2024-03-26T21:31:15.180111+0000","last_peered":"2024-03-26T21:31:15.180111+0000","last_clean":"2024-03-26T21:30:29.906831+0000","last_became_active":"2024-03-26T21:30:31.738596+0000","last_became_peered":"2024-03-26T21:30:31.738596+0000","last_unstale":"2024-03-26T21:31:15.180111+0000","last_undegraded":"2024-03-26T21:30:31.733688+0000","last_fullsized":"2024-03-26T21:30:31.733580+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:10.722203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.722203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:37.288268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":101,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.175788+0000","last_change":"2024-03-26T21:30:31.740769+0000","last_active":"2024-03-26T21:31:15.175788+0000","last_peered":"2024-03-26T21:31:15.175788+0000","last_clean":"2024-03-26T21:30:19.055952+0000","last_became_active":"2024-03-26T21:30:31.740769+0000","last_became_peered":"2024-03-26T21:30:31.740769+0000","last_unstale":"2024-03-26T21:31:15.175788+0000","last_undegraded":"2024-03-26T21:30:31.730676+0000","last_fullsized":"2024-03-26T21:30:31.730563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.058714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.153791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184059+0000","last_change":"2024-03-26T21:30:37.614796+0000","last_active":"2024-03-26T21:31:15.184059+0000","last_peered":"2024-03-26T21:31:15.184059+0000","last_clean":"2024-03-26T21:31:15.184059+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:31:15.184059+0000","last_undegraded":"2024-03-26T21:31:15.184059+0000","last_fullsized":"2024-03-26T21:31:15.184059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:37.614664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.614664+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:59.617494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.175274+0000","last_change":"2024-03-26T21:30:31.739460+0000","last_active":"2024-03-26T21:31:15.175274+0000","last_peered":"2024-03-26T21:31:15.175274+0000","last_clean":"2024-03-26T21:30:19.057225+0000","last_became_active":"2024-03-26T21:30:31.739460+0000","last_became_peered":"2024-03-26T21:30:31.739460+0000","last_unstale":"2024-03-26T21:31:15.175274+0000","last_undegraded":"2024-03-26T21:31:15.175274+0000","last_fullsized":"2024-03-26T21:30:31.733695+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184408+0000","last_change":"2024-03-26T21:29:56.787368+0000","last_active":"2024-03-26T21:31:15.184408+0000","last_peered":"2024-03-26T21:31:15.184408+0000","last_clean":"2024-03-26T21:31:15.184408+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:31:15.184408+0000","last_undegraded":"2024-03-26T21:31:15.184408+0000","last_fullsized":"2024-03-26T21:31:15.184408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.787260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.787260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.803062+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179748+0000","last_change":"2024-03-26T21:30:01.773999+0000","last_active":"2024-03-26T21:31:15.179748+0000","last_peered":"2024-03-26T21:31:15.179748+0000","last_clean":"2024-03-26T21:31:15.179748+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:31:15.179748+0000","last_undegraded":"2024-03-26T21:31:15.179748+0000","last_fullsized":"2024-03-26T21:31:15.179748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.773917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.773917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.422442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":111,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180136+0000","last_change":"2024-03-26T21:29:36.821598+0000","last_active":"2024-03-26T21:31:15.180136+0000","last_peered":"2024-03-26T21:31:15.180136+0000","last_clean":"2024-03-26T21:31:15.180136+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:31:15.180136+0000","last_undegraded":"2024-03-26T21:31:15.180136+0000","last_fullsized":"2024-03-26T21:31:15.180136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:29:36.821436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.821436+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177055+0000","last_change":"2024-03-26T21:30:31.738804+0000","last_active":"2024-03-26T21:31:15.177055+0000","last_peered":"2024-03-26T21:31:15.177055+0000","last_clean":"2024-03-26T21:30:19.144401+0000","last_became_active":"2024-03-26T21:30:31.738804+0000","last_became_peered":"2024-03-26T21:30:31.738804+0000","last_unstale":"2024-03-26T21:31:15.177055+0000","last_undegraded":"2024-03-26T21:31:15.177055+0000","last_fullsized":"2024-03-26T21:30:31.735055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.535399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.535399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:37.863776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179164+0000","last_change":"2024-03-26T21:30:31.739828+0000","last_active":"2024-03-26T21:31:15.179164+0000","last_peered":"2024-03-26T21:31:15.179164+0000","last_clean":"2024-03-26T21:30:19.905962+0000","last_became_active":"2024-03-26T21:30:31.739828+0000","last_became_peered":"2024-03-26T21:30:31.739828+0000","last_unstale":"2024-03-26T21:31:15.179164+0000","last_undegraded":"2024-03-26T21:31:15.179164+0000","last_fullsized":"2024-03-26T21:30:31.731999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.749046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.749046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:20.317347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184215+0000","last_change":"2024-03-26T21:29:52.893256+0000","last_active":"2024-03-26T21:31:15.184215+0000","last_peered":"2024-03-26T21:31:15.184215+0000","last_clean":"2024-03-26T21:31:15.184215+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:31:15.184215+0000","last_undegraded":"2024-03-26T21:31:15.184215+0000","last_fullsized":"2024-03-26T21:31:15.184215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.893078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.893078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.803030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179857+0000","last_change":"2024-03-26T21:29:57.736516+0000","last_active":"2024-03-26T21:31:15.179857+0000","last_peered":"2024-03-26T21:31:15.179857+0000","last_clean":"2024-03-26T21:31:15.179857+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:31:15.179857+0000","last_undegraded":"2024-03-26T21:31:15.179857+0000","last_fullsized":"2024-03-26T21:31:15.179857+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:29:57.736423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:57.736423+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180006+0000","last_change":"2024-03-26T21:30:29.844822+0000","last_active":"2024-03-26T21:31:15.180006+0000","last_peered":"2024-03-26T21:31:15.180006+0000","last_clean":"2024-03-26T21:31:15.180006+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:31:15.180006+0000","last_undegraded":"2024-03-26T21:31:15.180006+0000","last_fullsized":"2024-03-26T21:31:15.180006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:30:29.844710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:53.869088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177265+0000","last_change":"2024-03-26T21:30:31.742998+0000","last_active":"2024-03-26T21:31:15.177265+0000","last_peered":"2024-03-26T21:31:15.177265+0000","last_clean":"2024-03-26T21:30:19.144660+0000","last_became_active":"2024-03-26T21:30:31.742998+0000","last_became_peered":"2024-03-26T21:30:31.742998+0000","last_unstale":"2024-03-26T21:31:15.177265+0000","last_undegraded":"2024-03-26T21:31:15.177265+0000","last_fullsized":"2024-03-26T21:30:31.735078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.667895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169766+0000","last_change":"2024-03-26T21:30:01.424034+0000","last_active":"2024-03-26T21:31:15.169766+0000","last_peered":"2024-03-26T21:31:15.169766+0000","last_clean":"2024-03-26T21:31:15.169766+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:31:15.169766+0000","last_undegraded":"2024-03-26T21:31:15.169766+0000","last_fullsized":"2024-03-26T21:31:15.169766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:01.423920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.423920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.782127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":111,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179686+0000","last_change":"2024-03-26T21:30:18.768249+0000","last_active":"2024-03-26T21:31:15.179686+0000","last_peered":"2024-03-26T21:31:15.179686+0000","last_clean":"2024-03-26T21:31:15.179686+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:31:15.179686+0000","last_undegraded":"2024-03-26T21:31:15.179686+0000","last_fullsized":"2024-03-26T21:31:15.179686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:18.768133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:18.768133+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.279774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.182860+0000","last_change":"2024-03-26T21:28:40.889084+0000","last_active":"2024-03-26T21:31:15.182860+0000","last_peered":"2024-03-26T21:31:15.182860+0000","last_clean":"2024-03-26T21:31:15.182860+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:31:15.182860+0000","last_undegraded":"2024-03-26T21:31:15.182860+0000","last_fullsized":"2024-03-26T21:31:15.182860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179519+0000","last_change":"2024-03-26T21:30:04.703815+0000","last_active":"2024-03-26T21:31:15.179519+0000","last_peered":"2024-03-26T21:31:15.179519+0000","last_clean":"2024-03-26T21:31:15.179519+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:31:15.179519+0000","last_undegraded":"2024-03-26T21:31:15.179519+0000","last_fullsized":"2024-03-26T21:31:15.179519+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.703726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.703726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.874695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177657+0000","last_change":"2024-03-26T21:30:31.736143+0000","last_active":"2024-03-26T21:31:15.177657+0000","last_peered":"2024-03-26T21:31:15.177657+0000","last_clean":"2024-03-26T21:30:19.144685+0000","last_became_active":"2024-03-26T21:30:31.736143+0000","last_became_peered":"2024-03-26T21:30:31.736143+0000","last_unstale":"2024-03-26T21:31:15.177657+0000","last_undegraded":"2024-03-26T21:31:15.177657+0000","last_fullsized":"2024-03-26T21:30:31.733617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.668004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.174973+0000","last_change":"2024-03-26T21:30:10.096212+0000","last_active":"2024-03-26T21:31:15.174973+0000","last_peered":"2024-03-26T21:31:15.174973+0000","last_clean":"2024-03-26T21:31:15.174973+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:31:15.174973+0000","last_undegraded":"2024-03-26T21:31:15.174973+0000","last_fullsized":"2024-03-26T21:31:15.174973+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.096107+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.062377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179231+0000","last_change":"2024-03-26T21:30:11.623386+0000","last_active":"2024-03-26T21:31:15.179231+0000","last_peered":"2024-03-26T21:31:15.179231+0000","last_clean":"2024-03-26T21:31:15.179231+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:31:15.179231+0000","last_undegraded":"2024-03-26T21:31:15.179231+0000","last_fullsized":"2024-03-26T21:31:15.179231+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:11.623251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.623251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.174643+0000","last_change":"2024-03-26T21:30:04.095313+0000","last_active":"2024-03-26T21:31:15.174643+0000","last_peered":"2024-03-26T21:31:15.174643+0000","last_clean":"2024-03-26T21:31:15.174643+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:31:15.174643+0000","last_undegraded":"2024-03-26T21:31:15.174643+0000","last_fullsized":"2024-03-26T21:31:15.174643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.095208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.095208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.418775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":32,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177571+0000","last_change":"2024-03-26T21:30:31.741210+0000","last_active":"2024-03-26T21:31:15.177571+0000","last_peered":"2024-03-26T21:31:15.177571+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.741210+0000","last_became_peered":"2024-03-26T21:30:31.741210+0000","last_unstale":"2024-03-26T21:31:15.177571+0000","last_undegraded":"2024-03-26T21:31:15.177571+0000","last_fullsized":"2024-03-26T21:30:31.734566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:26.155650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:26.155650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:55.024760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":69,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.179573+0000","last_change":"2024-03-26T21:30:31.741165+0000","last_active":"2024-03-26T21:31:15.179573+0000","last_peered":"2024-03-26T21:31:15.179573+0000","last_clean":"2024-03-26T21:28:42.207254+0000","last_became_active":"2024-03-26T21:30:31.741165+0000","last_became_peered":"2024-03-26T21:30:31.741165+0000","last_unstale":"2024-03-26T21:31:15.179573+0000","last_undegraded":"2024-03-26T21:30:31.733400+0000","last_fullsized":"2024-03-26T21:30:31.733163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":85,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183252+0000","last_change":"2024-03-26T21:30:04.667815+0000","last_active":"2024-03-26T21:31:15.183252+0000","last_peered":"2024-03-26T21:31:15.183252+0000","last_clean":"2024-03-26T21:31:15.183252+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:31:15.183252+0000","last_undegraded":"2024-03-26T21:31:15.183252+0000","last_fullsized":"2024-03-26T21:31:15.183252+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:30:04.667736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.667736+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:23.712159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":130,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170047+0000","last_change":"2024-03-26T21:29:47.419090+0000","last_active":"2024-03-26T21:31:15.170047+0000","last_peered":"2024-03-26T21:31:15.170047+0000","last_clean":"2024-03-26T21:31:15.170047+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:31:15.170047+0000","last_undegraded":"2024-03-26T21:31:15.170047+0000","last_fullsized":"2024-03-26T21:31:15.170047+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:29:47.418949+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.589102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177386+0000","last_change":"2024-03-26T21:30:31.738173+0000","last_active":"2024-03-26T21:31:15.177386+0000","last_peered":"2024-03-26T21:31:15.177386+0000","last_clean":"2024-03-26T21:30:19.144537+0000","last_became_active":"2024-03-26T21:30:31.738173+0000","last_became_peered":"2024-03-26T21:30:31.738173+0000","last_unstale":"2024-03-26T21:31:15.177386+0000","last_undegraded":"2024-03-26T21:31:15.177386+0000","last_fullsized":"2024-03-26T21:30:31.733359+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.668018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180217+0000","last_change":"2024-03-26T21:29:55.733996+0000","last_active":"2024-03-26T21:31:15.180217+0000","last_peered":"2024-03-26T21:31:15.180217+0000","last_clean":"2024-03-26T21:31:15.180217+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:31:15.180217+0000","last_undegraded":"2024-03-26T21:31:15.180217+0000","last_fullsized":"2024-03-26T21:31:15.180217+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:29:55.733905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.733905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.874613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.174003+0000","last_change":"2024-03-26T21:30:31.744131+0000","last_active":"2024-03-26T21:31:15.174003+0000","last_peered":"2024-03-26T21:31:15.174003+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.744131+0000","last_became_peered":"2024-03-26T21:30:31.744131+0000","last_unstale":"2024-03-26T21:31:15.174003+0000","last_undegraded":"2024-03-26T21:31:15.174003+0000","last_fullsized":"2024-03-26T21:30:31.735473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:36.870940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":132,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180386+0000","last_change":"2024-03-26T21:30:37.766284+0000","last_active":"2024-03-26T21:31:15.180386+0000","last_peered":"2024-03-26T21:31:15.180386+0000","last_clean":"2024-03-26T21:31:15.180386+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:31:15.180386+0000","last_undegraded":"2024-03-26T21:31:15.180386+0000","last_fullsized":"2024-03-26T21:31:15.180386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:37.766196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.766196+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:07.457096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183694+0000","last_change":"2024-03-26T21:30:33.644993+0000","last_active":"2024-03-26T21:31:15.183694+0000","last_peered":"2024-03-26T21:31:15.183694+0000","last_clean":"2024-03-26T21:31:15.183694+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:31:15.183694+0000","last_undegraded":"2024-03-26T21:31:15.183694+0000","last_fullsized":"2024-03-26T21:31:15.183694+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:33.644876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:33.644876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.625879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":116,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170303+0000","last_change":"2024-03-26T21:31:14.634077+0000","last_active":"2024-03-26T21:31:15.170303+0000","last_peered":"2024-03-26T21:31:15.170303+0000","last_clean":"2024-03-26T21:31:15.170303+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:31:15.170303+0000","last_undegraded":"2024-03-26T21:31:15.170303+0000","last_fullsized":"2024-03-26T21:31:15.170303+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:31:14.633936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.633936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:30.906840+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.173663+0000","last_change":"2024-03-26T21:30:31.739981+0000","last_active":"2024-03-26T21:31:15.173663+0000","last_peered":"2024-03-26T21:31:15.173663+0000","last_clean":"2024-03-26T21:30:19.265055+0000","last_became_active":"2024-03-26T21:30:31.739981+0000","last_became_peered":"2024-03-26T21:30:31.739981+0000","last_unstale":"2024-03-26T21:31:15.173663+0000","last_undegraded":"2024-03-26T21:31:15.173663+0000","last_fullsized":"2024-03-26T21:30:31.736585+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.958658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.958658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.659424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.181465+0000","last_change":"2024-03-26T21:30:24.865430+0000","last_active":"2024-03-26T21:31:15.181465+0000","last_peered":"2024-03-26T21:31:15.181465+0000","last_clean":"2024-03-26T21:31:15.181465+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:31:15.181465+0000","last_undegraded":"2024-03-26T21:31:15.181465+0000","last_fullsized":"2024-03-26T21:31:15.181465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:24.865340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:24.865340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:53.374497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177082+0000","last_change":"2024-03-26T21:30:31.742874+0000","last_active":"2024-03-26T21:31:15.177082+0000","last_peered":"2024-03-26T21:31:15.177082+0000","last_clean":"2024-03-26T21:30:30.545271+0000","last_became_active":"2024-03-26T21:30:31.742874+0000","last_became_peered":"2024-03-26T21:30:31.742874+0000","last_unstale":"2024-03-26T21:31:15.177082+0000","last_undegraded":"2024-03-26T21:31:15.177082+0000","last_fullsized":"2024-03-26T21:30:31.735756+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:30.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:30.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:54.392834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":108,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184256+0000","last_change":"2024-03-26T21:30:00.719906+0000","last_active":"2024-03-26T21:31:15.184256+0000","last_peered":"2024-03-26T21:31:15.184256+0000","last_clean":"2024-03-26T21:31:15.184256+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:31:15.184256+0000","last_undegraded":"2024-03-26T21:31:15.184256+0000","last_fullsized":"2024-03-26T21:31:15.184256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:30:00.719791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:00.719791+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:22.215539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177647+0000","last_change":"2024-03-26T21:30:31.742256+0000","last_active":"2024-03-26T21:31:15.177647+0000","last_peered":"2024-03-26T21:31:15.177647+0000","last_clean":"2024-03-26T21:30:27.542711+0000","last_became_active":"2024-03-26T21:30:31.742256+0000","last_became_peered":"2024-03-26T21:30:31.742256+0000","last_unstale":"2024-03-26T21:31:15.177647+0000","last_undegraded":"2024-03-26T21:31:15.177647+0000","last_fullsized":"2024-03-26T21:30:31.733298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:30:27.542602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:55.327144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170166+0000","last_change":"2024-03-26T21:30:03.444638+0000","last_active":"2024-03-26T21:31:15.170166+0000","last_peered":"2024-03-26T21:31:15.170166+0000","last_clean":"2024-03-26T21:31:15.170166+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:31:15.170166+0000","last_undegraded":"2024-03-26T21:31:15.170166+0000","last_fullsized":"2024-03-26T21:31:15.170166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.444530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.444530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.589118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.178846+0000","last_change":"2024-03-26T21:29:53.937502+0000","last_active":"2024-03-26T21:31:15.178846+0000","last_peered":"2024-03-26T21:31:15.178846+0000","last_clean":"2024-03-26T21:31:15.178846+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:31:15.178846+0000","last_undegraded":"2024-03-26T21:31:15.178846+0000","last_fullsized":"2024-03-26T21:31:15.178846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:53.937386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:53.937386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":115,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.181126+0000","last_change":"2024-03-26T21:30:09.668099+0000","last_active":"2024-03-26T21:31:15.181126+0000","last_peered":"2024-03-26T21:31:15.181126+0000","last_clean":"2024-03-26T21:31:15.181126+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:31:15.181126+0000","last_undegraded":"2024-03-26T21:31:15.181126+0000","last_fullsized":"2024-03-26T21:31:15.181126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:30:09.668002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.668002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:30.607945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"105'2746","reported_seq":2776,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.292082+0000","last_change":"2024-03-26T21:30:38.544003+0000","last_active":"2024-03-26T21:31:16.292082+0000","last_peered":"2024-03-26T21:31:16.292082+0000","last_clean":"2024-03-26T21:31:16.292082+0000","last_became_active":"2024-03-26T21:30:38.543645+0000","last_became_peered":"2024-03-26T21:30:38.543645+0000","last_unstale":"2024-03-26T21:31:16.292082+0000","last_undegraded":"2024-03-26T21:31:16.292082+0000","last_fullsized":"2024-03-26T21:31:16.292082+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.233186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179961856,"num_objects":2746,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":175744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170499+0000","last_change":"2024-03-26T21:30:12.503906+0000","last_active":"2024-03-26T21:31:15.170499+0000","last_peered":"2024-03-26T21:31:15.170499+0000","last_clean":"2024-03-26T21:31:15.170499+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:31:15.170499+0000","last_undegraded":"2024-03-26T21:31:15.170499+0000","last_fullsized":"2024-03-26T21:31:15.170499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.503821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.503821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:42.279930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":109,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.178129+0000","last_change":"2024-03-26T21:30:31.738782+0000","last_active":"2024-03-26T21:31:15.178129+0000","last_peered":"2024-03-26T21:31:15.178129+0000","last_clean":"2024-03-26T21:30:19.145088+0000","last_became_active":"2024-03-26T21:30:31.738782+0000","last_became_peered":"2024-03-26T21:30:31.738782+0000","last_unstale":"2024-03-26T21:31:15.178129+0000","last_undegraded":"2024-03-26T21:30:31.736341+0000","last_fullsized":"2024-03-26T21:30:31.736205+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.667917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":117,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169314+0000","last_change":"2024-03-26T21:30:07.493378+0000","last_active":"2024-03-26T21:31:15.169314+0000","last_peered":"2024-03-26T21:31:15.169314+0000","last_clean":"2024-03-26T21:31:15.169314+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:31:15.169314+0000","last_undegraded":"2024-03-26T21:31:15.169314+0000","last_fullsized":"2024-03-26T21:31:15.169314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:07.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.493257+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:31.720965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":125,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.177286+0000","last_change":"2024-03-26T21:30:31.743316+0000","last_active":"2024-03-26T21:31:15.177286+0000","last_peered":"2024-03-26T21:31:15.177286+0000","last_clean":"2024-03-26T21:30:29.906838+0000","last_became_active":"2024-03-26T21:30:31.743316+0000","last_became_peered":"2024-03-26T21:30:31.743316+0000","last_unstale":"2024-03-26T21:31:15.177286+0000","last_undegraded":"2024-03-26T21:30:31.736202+0000","last_fullsized":"2024-03-26T21:30:31.736088+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.667973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"105'2781","reported_seq":2811,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.307558+0000","last_change":"2024-03-26T21:30:38.548354+0000","last_active":"2024-03-26T21:31:16.307558+0000","last_peered":"2024-03-26T21:31:16.307558+0000","last_clean":"2024-03-26T21:31:16.307558+0000","last_became_active":"2024-03-26T21:30:38.548054+0000","last_became_peered":"2024-03-26T21:30:38.548054+0000","last_unstale":"2024-03-26T21:31:16.307558+0000","last_undegraded":"2024-03-26T21:31:16.307558+0000","last_fullsized":"2024-03-26T21:31:16.307558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2781,"log_dups_size":0,"ondisk_log_size":2781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:42.833738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182255616,"num_objects":2781,"num_object_clones":0,"num_object_copies":8343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2781,"num_write_kb":177984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179710+0000","last_change":"2024-03-26T21:30:40.780018+0000","last_active":"2024-03-26T21:31:15.179710+0000","last_peered":"2024-03-26T21:31:15.179710+0000","last_clean":"2024-03-26T21:31:15.179710+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:31:15.179710+0000","last_undegraded":"2024-03-26T21:31:15.179710+0000","last_fullsized":"2024-03-26T21:31:15.179710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:40.779910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:40.779910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:10.033861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":106,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.177807+0000","last_change":"2024-03-26T21:30:31.744165+0000","last_active":"2024-03-26T21:31:15.177807+0000","last_peered":"2024-03-26T21:31:15.177807+0000","last_clean":"2024-03-26T21:30:19.145058+0000","last_became_active":"2024-03-26T21:30:31.744165+0000","last_became_peered":"2024-03-26T21:30:31.744165+0000","last_unstale":"2024-03-26T21:31:15.177807+0000","last_undegraded":"2024-03-26T21:30:31.735909+0000","last_fullsized":"2024-03-26T21:30:31.735789+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.667989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":73,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.176001+0000","last_change":"2024-03-26T21:30:31.740300+0000","last_active":"2024-03-26T21:31:15.176001+0000","last_peered":"2024-03-26T21:31:15.176001+0000","last_clean":"2024-03-26T21:28:42.051440+0000","last_became_active":"2024-03-26T21:30:31.740300+0000","last_became_peered":"2024-03-26T21:30:31.740300+0000","last_unstale":"2024-03-26T21:31:15.176001+0000","last_undegraded":"2024-03-26T21:30:31.730730+0000","last_fullsized":"2024-03-26T21:30:31.730592+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.153766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.174831+0000","last_change":"2024-03-26T21:30:02.905489+0000","last_active":"2024-03-26T21:31:15.174831+0000","last_peered":"2024-03-26T21:31:15.174831+0000","last_clean":"2024-03-26T21:31:15.174831+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:31:15.174831+0000","last_undegraded":"2024-03-26T21:31:15.174831+0000","last_fullsized":"2024-03-26T21:31:15.174831+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:02.905408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.905408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.664056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"105'2873","reported_seq":2899,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:16.328923+0000","last_change":"2024-03-26T21:30:53.719400+0000","last_active":"2024-03-26T21:31:16.328923+0000","last_peered":"2024-03-26T21:31:16.328923+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.546367+0000","last_became_peered":"2024-03-26T21:30:38.546367+0000","last_unstale":"2024-03-26T21:31:16.328923+0000","last_undegraded":"2024-03-26T21:30:53.709328+0000","last_fullsized":"2024-03-26T21:30:38.539071+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2873,"log_dups_size":0,"ondisk_log_size":2873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:54.304747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188284928,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2873,"num_write_kb":183872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2873}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.176942+0000","last_change":"2024-03-26T21:30:39.632228+0000","last_active":"2024-03-26T21:31:15.176942+0000","last_peered":"2024-03-26T21:31:15.176942+0000","last_clean":"2024-03-26T21:31:15.176942+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:31:15.176942+0000","last_undegraded":"2024-03-26T21:31:15.176942+0000","last_fullsized":"2024-03-26T21:31:15.176942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:39.632108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:39.632108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:54.704696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.176016+0000","last_change":"2024-03-26T21:30:03.938919+0000","last_active":"2024-03-26T21:31:15.176016+0000","last_peered":"2024-03-26T21:31:15.176016+0000","last_clean":"2024-03-26T21:31:15.176016+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:31:15.176016+0000","last_undegraded":"2024-03-26T21:31:15.176016+0000","last_fullsized":"2024-03-26T21:31:15.176016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.938836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.938836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177920+0000","last_change":"2024-03-26T21:29:48.625718+0000","last_active":"2024-03-26T21:31:15.177920+0000","last_peered":"2024-03-26T21:31:15.177920+0000","last_clean":"2024-03-26T21:31:15.177920+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:31:15.177920+0000","last_undegraded":"2024-03-26T21:31:15.177920+0000","last_fullsized":"2024-03-26T21:31:15.177920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.625595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.625595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:15.668053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179429+0000","last_change":"2024-03-26T21:29:58.792883+0000","last_active":"2024-03-26T21:31:15.179429+0000","last_peered":"2024-03-26T21:31:15.179429+0000","last_clean":"2024-03-26T21:31:15.179429+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:31:15.179429+0000","last_undegraded":"2024-03-26T21:31:15.179429+0000","last_fullsized":"2024-03-26T21:31:15.179429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:58.792783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.792783+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"105'2855","reported_seq":2882,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:16.323598+0000","last_change":"2024-03-26T21:30:53.710423+0000","last_active":"2024-03-26T21:31:16.323598+0000","last_peered":"2024-03-26T21:31:16.323598+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.540186+0000","last_became_peered":"2024-03-26T21:30:38.540186+0000","last_unstale":"2024-03-26T21:31:16.323598+0000","last_undegraded":"2024-03-26T21:30:53.707983+0000","last_fullsized":"2024-03-26T21:30:38.534077+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2855,"log_dups_size":0,"ondisk_log_size":2855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:41.703315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2855,"num_write_kb":182720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":2855}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.184947+0000","last_change":"2024-03-26T21:30:08.751794+0000","last_active":"2024-03-26T21:31:15.184947+0000","last_peered":"2024-03-26T21:31:15.184947+0000","last_clean":"2024-03-26T21:31:15.184947+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:31:15.184947+0000","last_undegraded":"2024-03-26T21:31:15.184947+0000","last_fullsized":"2024-03-26T21:31:15.184947+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.751700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:08.751700+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.751700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.625207+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.177239+0000","last_change":"2024-03-26T21:30:31.741930+0000","last_active":"2024-03-26T21:31:15.177239+0000","last_peered":"2024-03-26T21:31:15.177239+0000","last_clean":"2024-03-26T21:30:19.264875+0000","last_became_active":"2024-03-26T21:30:31.741930+0000","last_became_peered":"2024-03-26T21:30:31.741930+0000","last_unstale":"2024-03-26T21:31:15.177239+0000","last_undegraded":"2024-03-26T21:31:15.177239+0000","last_fullsized":"2024-03-26T21:30:31.734873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.110332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169787+0000","last_change":"2024-03-26T21:30:16.534007+0000","last_active":"2024-03-26T21:31:15.169787+0000","last_peered":"2024-03-26T21:31:15.169787+0000","last_clean":"2024-03-26T21:31:15.169787+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:31:15.169787+0000","last_undegraded":"2024-03-26T21:31:15.169787+0000","last_fullsized":"2024-03-26T21:31:15.169787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:16.533899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:16.533899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.535048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":103,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.184136+0000","last_change":"2024-03-26T21:30:31.747569+0000","last_active":"2024-03-26T21:31:15.184136+0000","last_peered":"2024-03-26T21:31:15.184136+0000","last_clean":"2024-03-26T21:30:19.056675+0000","last_became_active":"2024-03-26T21:30:31.747569+0000","last_became_peered":"2024-03-26T21:30:31.747569+0000","last_unstale":"2024-03-26T21:31:15.184136+0000","last_undegraded":"2024-03-26T21:30:31.738166+0000","last_fullsized":"2024-03-26T21:30:31.738045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"105'2902","reported_seq":2932,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.278989+0000","last_change":"2024-03-26T21:30:38.549130+0000","last_active":"2024-03-26T21:31:16.278989+0000","last_peered":"2024-03-26T21:31:16.278989+0000","last_clean":"2024-03-26T21:31:16.278989+0000","last_became_active":"2024-03-26T21:30:38.548880+0000","last_became_peered":"2024-03-26T21:30:38.548880+0000","last_unstale":"2024-03-26T21:31:16.278989+0000","last_undegraded":"2024-03-26T21:31:16.278989+0000","last_fullsized":"2024-03-26T21:31:16.278989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:42.209599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":185728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.180570+0000","last_change":"2024-03-26T21:30:31.738178+0000","last_active":"2024-03-26T21:31:15.180570+0000","last_peered":"2024-03-26T21:31:15.180570+0000","last_clean":"2024-03-26T21:30:19.792003+0000","last_became_active":"2024-03-26T21:30:31.738178+0000","last_became_peered":"2024-03-26T21:30:31.738178+0000","last_unstale":"2024-03-26T21:31:15.180570+0000","last_undegraded":"2024-03-26T21:31:15.180570+0000","last_fullsized":"2024-03-26T21:30:31.731810+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170263+0000","last_change":"2024-03-26T21:30:06.496381+0000","last_active":"2024-03-26T21:31:15.170263+0000","last_peered":"2024-03-26T21:31:15.170263+0000","last_clean":"2024-03-26T21:31:15.170263+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:31:15.170263+0000","last_undegraded":"2024-03-26T21:31:15.170263+0000","last_fullsized":"2024-03-26T21:31:15.170263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:30:06.496257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.496257+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:28.713708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.184383+0000","last_change":"2024-03-26T21:30:31.740606+0000","last_active":"2024-03-26T21:31:15.184383+0000","last_peered":"2024-03-26T21:31:15.184383+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740606+0000","last_became_peered":"2024-03-26T21:30:31.740606+0000","last_unstale":"2024-03-26T21:31:15.184383+0000","last_undegraded":"2024-03-26T21:31:15.184383+0000","last_fullsized":"2024-03-26T21:30:31.737530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:15.290476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:15.290476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:31.741087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":127,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170135+0000","last_change":"2024-03-26T21:29:56.452291+0000","last_active":"2024-03-26T21:31:15.170135+0000","last_peered":"2024-03-26T21:31:15.170135+0000","last_clean":"2024-03-26T21:31:15.170135+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:31:15.170135+0000","last_undegraded":"2024-03-26T21:31:15.170135+0000","last_fullsized":"2024-03-26T21:31:15.170135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:29:56.452170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.452170+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.183613+0000","last_change":"2024-03-26T21:30:31.745830+0000","last_active":"2024-03-26T21:31:15.183613+0000","last_peered":"2024-03-26T21:31:15.183613+0000","last_clean":"2024-03-26T21:30:19.056424+0000","last_became_active":"2024-03-26T21:30:31.745830+0000","last_became_peered":"2024-03-26T21:30:31.745830+0000","last_unstale":"2024-03-26T21:31:15.183613+0000","last_undegraded":"2024-03-26T21:31:15.183613+0000","last_fullsized":"2024-03-26T21:30:31.736422+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.721880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.721880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.703351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"105'2768","reported_seq":2795,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:16.323797+0000","last_change":"2024-03-26T21:30:53.720906+0000","last_active":"2024-03-26T21:31:16.323797+0000","last_peered":"2024-03-26T21:31:16.323797+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.544117+0000","last_became_peered":"2024-03-26T21:30:38.544117+0000","last_unstale":"2024-03-26T21:31:16.323797+0000","last_undegraded":"2024-03-26T21:30:53.719381+0000","last_fullsized":"2024-03-26T21:30:38.538732+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:52.670273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":177152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.178022+0000","last_change":"2024-03-26T21:30:14.531043+0000","last_active":"2024-03-26T21:31:15.178022+0000","last_peered":"2024-03-26T21:31:15.178022+0000","last_clean":"2024-03-26T21:31:15.178022+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:31:15.178022+0000","last_undegraded":"2024-03-26T21:31:15.178022+0000","last_fullsized":"2024-03-26T21:31:15.178022+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:14.530923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:14.530923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:35.221099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177926+0000","last_change":"2024-03-26T21:29:36.249432+0000","last_active":"2024-03-26T21:31:15.177926+0000","last_peered":"2024-03-26T21:31:15.177926+0000","last_clean":"2024-03-26T21:31:15.177926+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:31:15.177926+0000","last_undegraded":"2024-03-26T21:31:15.177926+0000","last_fullsized":"2024-03-26T21:31:15.177926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:36.249302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:36.249302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.153709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":103,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173011+0000","last_change":"2024-03-26T21:29:41.695430+0000","last_active":"2024-03-26T21:31:15.173011+0000","last_peered":"2024-03-26T21:31:15.173011+0000","last_clean":"2024-03-26T21:31:15.173011+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:31:15.173011+0000","last_undegraded":"2024-03-26T21:31:15.173011+0000","last_fullsized":"2024-03-26T21:31:15.173011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.695322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.695322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":124,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173434+0000","last_change":"2024-03-26T21:29:55.054708+0000","last_active":"2024-03-26T21:31:15.173434+0000","last_peered":"2024-03-26T21:31:15.173434+0000","last_clean":"2024-03-26T21:31:15.173434+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:31:15.173434+0000","last_undegraded":"2024-03-26T21:31:15.173434+0000","last_fullsized":"2024-03-26T21:31:15.173434+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.054601+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.383564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173993+0000","last_change":"2024-03-26T21:30:06.957057+0000","last_active":"2024-03-26T21:31:15.173993+0000","last_peered":"2024-03-26T21:31:15.173993+0000","last_clean":"2024-03-26T21:31:15.173993+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:31:15.173993+0000","last_undegraded":"2024-03-26T21:31:15.173993+0000","last_fullsized":"2024-03-26T21:31:15.173993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.956977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.956977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"105'2826","reported_seq":2856,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.329277+0000","last_change":"2024-03-26T21:30:38.551063+0000","last_active":"2024-03-26T21:31:16.329277+0000","last_peered":"2024-03-26T21:31:16.329277+0000","last_clean":"2024-03-26T21:31:16.329277+0000","last_became_active":"2024-03-26T21:30:38.550658+0000","last_became_peered":"2024-03-26T21:30:38.550658+0000","last_unstale":"2024-03-26T21:31:16.329277+0000","last_undegraded":"2024-03-26T21:31:16.329277+0000","last_fullsized":"2024-03-26T21:31:16.329277+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2826,"log_dups_size":0,"ondisk_log_size":2826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:49.737850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":8478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":180864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.170443+0000","last_change":"2024-03-26T21:30:31.744917+0000","last_active":"2024-03-26T21:31:15.170443+0000","last_peered":"2024-03-26T21:31:15.170443+0000","last_clean":"2024-03-26T21:30:19.907127+0000","last_became_active":"2024-03-26T21:30:31.744917+0000","last_became_peered":"2024-03-26T21:30:31.744917+0000","last_unstale":"2024-03-26T21:31:15.170443+0000","last_undegraded":"2024-03-26T21:31:15.170443+0000","last_fullsized":"2024-03-26T21:30:31.734562+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.472985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.472985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.589081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":347,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177075+0000","last_change":"2024-03-26T21:30:32.576223+0000","last_active":"2024-03-26T21:31:15.177075+0000","last_peered":"2024-03-26T21:31:15.177075+0000","last_clean":"2024-03-26T21:31:15.177075+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:31:15.177075+0000","last_undegraded":"2024-03-26T21:31:15.177075+0000","last_fullsized":"2024-03-26T21:31:15.177075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:30:32.576135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:30:32.576135+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:53.692684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.181041+0000","last_change":"2024-03-26T21:31:14.873351+0000","last_active":"2024-03-26T21:31:15.181041+0000","last_peered":"2024-03-26T21:31:15.181041+0000","last_clean":"2024-03-26T21:31:15.181041+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:31:15.181041+0000","last_undegraded":"2024-03-26T21:31:15.181041+0000","last_fullsized":"2024-03-26T21:31:15.181041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:14.872965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.872965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.836988+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"94'25","reported_seq":159,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.180012+0000","last_change":"2024-03-26T21:30:31.737537+0000","last_active":"2024-03-26T21:31:15.180012+0000","last_peered":"2024-03-26T21:31:15.180012+0000","last_clean":"2024-03-26T21:30:29.907049+0000","last_became_active":"2024-03-26T21:30:31.737537+0000","last_became_peered":"2024-03-26T21:30:31.737537+0000","last_unstale":"2024-03-26T21:31:15.180012+0000","last_undegraded":"2024-03-26T21:30:31.733102+0000","last_fullsized":"2024-03-26T21:30:31.732979+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473777+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.752983+0000","last_change":"2024-03-26T21:31:15.753021+0000","last_active":"2024-03-26T21:31:15.752983+0000","last_peered":"2024-03-26T21:31:15.752983+0000","last_clean":"2024-03-26T21:31:15.752983+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:31:15.752983+0000","last_undegraded":"2024-03-26T21:31:15.752983+0000","last_fullsized":"2024-03-26T21:31:15.752983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:15.752907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:15.752907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:20.838766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"105'2829","reported_seq":2858,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.301139+0000","last_change":"2024-03-26T21:30:38.543845+0000","last_active":"2024-03-26T21:31:16.301139+0000","last_peered":"2024-03-26T21:31:16.301139+0000","last_clean":"2024-03-26T21:31:16.301139+0000","last_became_active":"2024-03-26T21:30:38.543531+0000","last_became_peered":"2024-03-26T21:30:38.543531+0000","last_unstale":"2024-03-26T21:31:16.301139+0000","last_undegraded":"2024-03-26T21:31:16.301139+0000","last_fullsized":"2024-03-26T21:31:16.301139+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:43.805226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":181056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173344+0000","last_change":"2024-03-26T21:30:16.082361+0000","last_active":"2024-03-26T21:31:15.173344+0000","last_peered":"2024-03-26T21:31:15.173344+0000","last_clean":"2024-03-26T21:31:15.173344+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:31:15.173344+0000","last_undegraded":"2024-03-26T21:31:15.173344+0000","last_fullsized":"2024-03-26T21:31:15.173344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:16.082196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:16.082196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:30.462838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.185166+0000","last_change":"2024-03-26T21:30:31.743896+0000","last_active":"2024-03-26T21:31:15.185166+0000","last_peered":"2024-03-26T21:31:15.185166+0000","last_clean":"2024-03-26T21:30:19.055870+0000","last_became_active":"2024-03-26T21:30:31.743896+0000","last_became_peered":"2024-03-26T21:30:31.743896+0000","last_unstale":"2024-03-26T21:31:15.185166+0000","last_undegraded":"2024-03-26T21:31:15.185166+0000","last_fullsized":"2024-03-26T21:30:31.733630+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:30:17.708355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:30.313183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":261,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177021+0000","last_change":"2024-03-26T21:30:23.578255+0000","last_active":"2024-03-26T21:31:15.177021+0000","last_peered":"2024-03-26T21:31:15.177021+0000","last_clean":"2024-03-26T21:31:15.177021+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:31:15.177021+0000","last_undegraded":"2024-03-26T21:31:15.177021+0000","last_fullsized":"2024-03-26T21:31:15.177021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:30:23.578172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:23.578172+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:29.515226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":130,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183366+0000","last_change":"2024-03-26T21:29:38.920799+0000","last_active":"2024-03-26T21:31:15.183366+0000","last_peered":"2024-03-26T21:31:15.183366+0000","last_clean":"2024-03-26T21:31:15.183366+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:31:15.183366+0000","last_undegraded":"2024-03-26T21:31:15.183366+0000","last_fullsized":"2024-03-26T21:31:15.183366+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:38.920701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:38.920701+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170684+0000","last_change":"2024-03-26T21:30:37.633283+0000","last_active":"2024-03-26T21:31:15.170684+0000","last_peered":"2024-03-26T21:31:15.170684+0000","last_clean":"2024-03-26T21:31:15.170684+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:31:15.170684+0000","last_undegraded":"2024-03-26T21:31:15.170684+0000","last_fullsized":"2024-03-26T21:31:15.170684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.633179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.633179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:38.963924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"105'2798","reported_seq":2828,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.292973+0000","last_change":"2024-03-26T21:30:38.544124+0000","last_active":"2024-03-26T21:31:16.292973+0000","last_peered":"2024-03-26T21:31:16.292973+0000","last_clean":"2024-03-26T21:31:16.292973+0000","last_became_active":"2024-03-26T21:30:38.543747+0000","last_became_peered":"2024-03-26T21:30:38.543747+0000","last_unstale":"2024-03-26T21:31:16.292973+0000","last_undegraded":"2024-03-26T21:31:16.292973+0000","last_fullsized":"2024-03-26T21:31:16.292973+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:45.006221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":179072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173919+0000","last_change":"2024-03-26T21:30:03.078966+0000","last_active":"2024-03-26T21:31:15.173919+0000","last_peered":"2024-03-26T21:31:15.173919+0000","last_clean":"2024-03-26T21:31:15.173919+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:31:15.173919+0000","last_undegraded":"2024-03-26T21:31:15.173919+0000","last_fullsized":"2024-03-26T21:31:15.173919+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.078885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.078885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:20.097118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.173609+0000","last_change":"2024-03-26T21:29:42.167843+0000","last_active":"2024-03-26T21:31:15.173609+0000","last_peered":"2024-03-26T21:31:15.173609+0000","last_clean":"2024-03-26T21:31:15.173609+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:31:15.173609+0000","last_undegraded":"2024-03-26T21:31:15.173609+0000","last_fullsized":"2024-03-26T21:31:15.173609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.167719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.153737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":128,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183994+0000","last_change":"2024-03-26T21:29:51.863137+0000","last_active":"2024-03-26T21:31:15.183994+0000","last_peered":"2024-03-26T21:31:15.183994+0000","last_clean":"2024-03-26T21:31:15.183994+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:31:15.183994+0000","last_undegraded":"2024-03-26T21:31:15.183994+0000","last_fullsized":"2024-03-26T21:31:15.183994+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:29:51.862960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.862960+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.803097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179530+0000","last_change":"2024-03-26T21:29:52.956781+0000","last_active":"2024-03-26T21:31:15.179530+0000","last_peered":"2024-03-26T21:31:15.179530+0000","last_clean":"2024-03-26T21:31:15.179530+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:31:15.179530+0000","last_undegraded":"2024-03-26T21:31:15.179530+0000","last_fullsized":"2024-03-26T21:31:15.179530+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:52.956669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.956669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"105'2738","reported_seq":2765,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:16.296163+0000","last_change":"2024-03-26T21:30:53.715187+0000","last_active":"2024-03-26T21:31:16.296163+0000","last_peered":"2024-03-26T21:31:16.296163+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.547364+0000","last_became_peered":"2024-03-26T21:30:38.547364+0000","last_unstale":"2024-03-26T21:31:16.296163+0000","last_undegraded":"2024-03-26T21:30:53.713329+0000","last_fullsized":"2024-03-26T21:30:38.537882+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2738,"log_dups_size":0,"ondisk_log_size":2738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.381586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179437568,"num_objects":2738,"num_object_clones":0,"num_object_copies":8214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":175232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183005+0000","last_change":"2024-03-26T21:30:12.703236+0000","last_active":"2024-03-26T21:31:15.183005+0000","last_peered":"2024-03-26T21:31:15.183005+0000","last_clean":"2024-03-26T21:31:15.183005+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:31:15.183005+0000","last_undegraded":"2024-03-26T21:31:15.183005+0000","last_fullsized":"2024-03-26T21:31:15.183005+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.703154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.703154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:42.163984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.170150+0000","last_change":"2024-03-26T21:30:38.590322+0000","last_active":"2024-03-26T21:31:15.170150+0000","last_peered":"2024-03-26T21:31:15.170150+0000","last_clean":"2024-03-26T21:31:15.170150+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:31:15.170150+0000","last_undegraded":"2024-03-26T21:31:15.170150+0000","last_fullsized":"2024-03-26T21:31:15.170150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:38.590254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:38.590254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:07.324127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":110,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.174725+0000","last_change":"2024-03-26T21:30:31.739773+0000","last_active":"2024-03-26T21:31:15.174725+0000","last_peered":"2024-03-26T21:31:15.174725+0000","last_clean":"2024-03-26T21:30:19.056603+0000","last_became_active":"2024-03-26T21:30:31.739773+0000","last_became_peered":"2024-03-26T21:30:31.739773+0000","last_unstale":"2024-03-26T21:31:15.174725+0000","last_undegraded":"2024-03-26T21:30:31.733621+0000","last_fullsized":"2024-03-26T21:30:31.733509+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:02.076064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.076064+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.733950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.183613+0000","last_change":"2024-03-26T21:30:11.661994+0000","last_active":"2024-03-26T21:31:15.183613+0000","last_peered":"2024-03-26T21:31:15.183613+0000","last_clean":"2024-03-26T21:31:15.183613+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:31:15.183613+0000","last_undegraded":"2024-03-26T21:31:15.183613+0000","last_fullsized":"2024-03-26T21:31:15.183613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.661905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.661905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:35.500861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"105'2770","reported_seq":2798,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.304854+0000","last_change":"2024-03-26T21:30:38.550210+0000","last_active":"2024-03-26T21:31:16.304854+0000","last_peered":"2024-03-26T21:31:16.304854+0000","last_clean":"2024-03-26T21:31:16.304854+0000","last_became_active":"2024-03-26T21:30:38.549806+0000","last_became_peered":"2024-03-26T21:30:38.549806+0000","last_unstale":"2024-03-26T21:31:16.304854+0000","last_undegraded":"2024-03-26T21:31:16.304854+0000","last_fullsized":"2024-03-26T21:31:16.304854+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:04.813813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":177280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.177452+0000","last_change":"2024-03-26T21:30:38.599138+0000","last_active":"2024-03-26T21:31:15.177452+0000","last_peered":"2024-03-26T21:31:15.177452+0000","last_clean":"2024-03-26T21:31:15.177452+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:31:15.177452+0000","last_undegraded":"2024-03-26T21:31:15.177452+0000","last_fullsized":"2024-03-26T21:31:15.177452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:38.599066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:38.599066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:51.556021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":92,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.183473+0000","last_change":"2024-03-26T21:30:31.746821+0000","last_active":"2024-03-26T21:31:15.183473+0000","last_peered":"2024-03-26T21:31:15.183473+0000","last_clean":"2024-03-26T21:30:19.056856+0000","last_became_active":"2024-03-26T21:30:31.746821+0000","last_became_peered":"2024-03-26T21:30:31.746821+0000","last_unstale":"2024-03-26T21:31:15.183473+0000","last_undegraded":"2024-03-26T21:30:31.736673+0000","last_fullsized":"2024-03-26T21:30:31.736528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":71,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.169364+0000","last_change":"2024-03-26T21:30:31.739394+0000","last_active":"2024-03-26T21:31:15.169364+0000","last_peered":"2024-03-26T21:31:15.169364+0000","last_clean":"2024-03-26T21:28:42.253768+0000","last_became_active":"2024-03-26T21:30:31.739394+0000","last_became_peered":"2024-03-26T21:30:31.739394+0000","last_unstale":"2024-03-26T21:31:15.169364+0000","last_undegraded":"2024-03-26T21:30:31.732454+0000","last_fullsized":"2024-03-26T21:30:31.732332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.098977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"94'9","reported_seq":133,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.180481+0000","last_change":"2024-03-26T21:30:31.745653+0000","last_active":"2024-03-26T21:31:15.180481+0000","last_peered":"2024-03-26T21:31:15.180481+0000","last_clean":"2024-03-26T21:30:29.906563+0000","last_became_active":"2024-03-26T21:30:31.745653+0000","last_became_peered":"2024-03-26T21:30:31.745653+0000","last_unstale":"2024-03-26T21:31:15.180481+0000","last_undegraded":"2024-03-26T21:30:31.730030+0000","last_fullsized":"2024-03-26T21:30:31.729924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:29:59.694434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:59.694434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:20.319790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"105'2830","reported_seq":2857,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:16.317358+0000","last_change":"2024-03-26T21:30:53.715233+0000","last_active":"2024-03-26T21:31:16.317358+0000","last_peered":"2024-03-26T21:31:16.317358+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.548505+0000","last_became_peered":"2024-03-26T21:30:38.548505+0000","last_unstale":"2024-03-26T21:31:16.317358+0000","last_undegraded":"2024-03-26T21:30:53.710995+0000","last_fullsized":"2024-03-26T21:30:38.537548+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:00.351929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185466880,"num_objects":2830,"num_object_clones":0,"num_object_copies":8490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2830,"num_write_kb":181120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.181594+0000","last_change":"2024-03-26T21:30:31.738939+0000","last_active":"2024-03-26T21:31:15.181594+0000","last_peered":"2024-03-26T21:31:15.181594+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738939+0000","last_became_peered":"2024-03-26T21:30:31.738939+0000","last_unstale":"2024-03-26T21:31:15.181594+0000","last_undegraded":"2024-03-26T21:31:15.181594+0000","last_fullsized":"2024-03-26T21:30:31.730286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180978+0000","last_change":"2024-03-26T21:29:56.690968+0000","last_active":"2024-03-26T21:31:15.180978+0000","last_peered":"2024-03-26T21:31:15.180978+0000","last_clean":"2024-03-26T21:31:15.180978+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:31:15.180978+0000","last_undegraded":"2024-03-26T21:31:15.180978+0000","last_fullsized":"2024-03-26T21:31:15.180978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.690884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.690884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:26.513440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":124,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.180369+0000","last_change":"2024-03-26T21:30:35.771996+0000","last_active":"2024-03-26T21:31:15.180369+0000","last_peered":"2024-03-26T21:31:15.180369+0000","last_clean":"2024-03-26T21:31:15.180369+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:31:15.180369+0000","last_undegraded":"2024-03-26T21:31:15.180369+0000","last_fullsized":"2024-03-26T21:31:15.180369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:30:35.771902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:35.771902+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:02.925148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179503+0000","last_change":"2024-03-26T21:30:31.738110+0000","last_active":"2024-03-26T21:31:15.179503+0000","last_peered":"2024-03-26T21:31:15.179503+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.738110+0000","last_became_peered":"2024-03-26T21:30:31.738110+0000","last_unstale":"2024-03-26T21:31:15.179503+0000","last_undegraded":"2024-03-26T21:31:15.179503+0000","last_fullsized":"2024-03-26T21:30:31.729956+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"105'2717","reported_seq":2747,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.282745+0000","last_change":"2024-03-26T21:30:38.540136+0000","last_active":"2024-03-26T21:31:16.282745+0000","last_peered":"2024-03-26T21:31:16.282745+0000","last_clean":"2024-03-26T21:31:16.282745+0000","last_became_active":"2024-03-26T21:30:38.539838+0000","last_became_peered":"2024-03-26T21:30:38.539838+0000","last_unstale":"2024-03-26T21:31:16.282745+0000","last_undegraded":"2024-03-26T21:31:16.282745+0000","last_fullsized":"2024-03-26T21:31:16.282745+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2717,"log_dups_size":0,"ondisk_log_size":2717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:51.332174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178061312,"num_objects":2717,"num_object_clones":0,"num_object_copies":8151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2717,"num_write_kb":173888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.169751+0000","last_change":"2024-03-26T21:30:04.491878+0000","last_active":"2024-03-26T21:31:15.169751+0000","last_peered":"2024-03-26T21:31:15.169751+0000","last_clean":"2024-03-26T21:31:15.169751+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:31:15.169751+0000","last_undegraded":"2024-03-26T21:31:15.169751+0000","last_fullsized":"2024-03-26T21:31:15.169751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.491791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.670339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"94'9","reported_seq":66,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.169862+0000","last_change":"2024-03-26T21:30:31.738518+0000","last_active":"2024-03-26T21:31:15.169862+0000","last_peered":"2024-03-26T21:31:15.169862+0000","last_clean":"2024-03-26T21:28:42.046048+0000","last_became_active":"2024-03-26T21:30:31.738518+0000","last_became_peered":"2024-03-26T21:30:31.738518+0000","last_unstale":"2024-03-26T21:31:15.169862+0000","last_undegraded":"2024-03-26T21:30:31.734182+0000","last_fullsized":"2024-03-26T21:30:31.734060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.407216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.407216+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.319914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179955+0000","last_change":"2024-03-26T21:29:41.689438+0000","last_active":"2024-03-26T21:31:15.179955+0000","last_peered":"2024-03-26T21:31:15.179955+0000","last_clean":"2024-03-26T21:31:15.179955+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:31:15.179955+0000","last_undegraded":"2024-03-26T21:31:15.179955+0000","last_fullsized":"2024-03-26T21:31:15.179955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:41.689332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:41.689332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.874642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":120,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.173602+0000","last_change":"2024-03-26T21:30:31.746320+0000","last_active":"2024-03-26T21:31:15.173602+0000","last_peered":"2024-03-26T21:31:15.173602+0000","last_clean":"2024-03-26T21:30:19.057783+0000","last_became_active":"2024-03-26T21:30:31.746320+0000","last_became_peered":"2024-03-26T21:30:31.746320+0000","last_unstale":"2024-03-26T21:31:15.173602+0000","last_undegraded":"2024-03-26T21:30:31.733035+0000","last_fullsized":"2024-03-26T21:30:31.732940+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:11.086876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.086876+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"105'2804","reported_seq":2834,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.279269+0000","last_change":"2024-03-26T21:30:38.544298+0000","last_active":"2024-03-26T21:31:16.279269+0000","last_peered":"2024-03-26T21:31:16.279269+0000","last_clean":"2024-03-26T21:31:16.279269+0000","last_became_active":"2024-03-26T21:30:38.543981+0000","last_became_peered":"2024-03-26T21:30:38.543981+0000","last_unstale":"2024-03-26T21:31:16.279269+0000","last_undegraded":"2024-03-26T21:31:16.279269+0000","last_fullsized":"2024-03-26T21:31:16.279269+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2804,"log_dups_size":0,"ondisk_log_size":2804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:42.580648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2804,"num_write_kb":179456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.179414+0000","last_change":"2024-03-26T21:30:12.655789+0000","last_active":"2024-03-26T21:31:15.179414+0000","last_peered":"2024-03-26T21:31:15.179414+0000","last_clean":"2024-03-26T21:31:15.179414+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:31:15.179414+0000","last_undegraded":"2024-03-26T21:31:15.179414+0000","last_fullsized":"2024-03-26T21:31:15.179414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.655702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:12.655702+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.655702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.170640+0000","last_change":"2024-03-26T21:30:31.742736+0000","last_active":"2024-03-26T21:31:15.170640+0000","last_peered":"2024-03-26T21:31:15.170640+0000","last_clean":"2024-03-26T21:30:19.907576+0000","last_became_active":"2024-03-26T21:30:31.742736+0000","last_became_peered":"2024-03-26T21:30:31.742736+0000","last_unstale":"2024-03-26T21:31:15.170640+0000","last_undegraded":"2024-03-26T21:31:15.170640+0000","last_fullsized":"2024-03-26T21:30:31.735097+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.589070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.172825+0000","last_change":"2024-03-26T21:30:31.738606+0000","last_active":"2024-03-26T21:31:15.172825+0000","last_peered":"2024-03-26T21:31:15.172825+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.738606+0000","last_became_peered":"2024-03-26T21:30:31.738606+0000","last_unstale":"2024-03-26T21:31:15.172825+0000","last_undegraded":"2024-03-26T21:31:15.172825+0000","last_fullsized":"2024-03-26T21:30:31.734264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:18.135205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":69,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.179976+0000","last_change":"2024-03-26T21:30:31.741438+0000","last_active":"2024-03-26T21:31:15.179976+0000","last_peered":"2024-03-26T21:31:15.179976+0000","last_clean":"2024-03-26T21:28:42.249232+0000","last_became_active":"2024-03-26T21:30:31.741438+0000","last_became_peered":"2024-03-26T21:30:31.741438+0000","last_unstale":"2024-03-26T21:31:15.179976+0000","last_undegraded":"2024-03-26T21:30:31.733158+0000","last_fullsized":"2024-03-26T21:30:31.733038+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"105'2823","reported_seq":2853,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.312887+0000","last_change":"2024-03-26T21:30:38.545760+0000","last_active":"2024-03-26T21:31:16.312887+0000","last_peered":"2024-03-26T21:31:16.312887+0000","last_clean":"2024-03-26T21:31:16.312887+0000","last_became_active":"2024-03-26T21:30:38.545485+0000","last_became_peered":"2024-03-26T21:30:38.545485+0000","last_unstale":"2024-03-26T21:31:16.312887+0000","last_undegraded":"2024-03-26T21:31:16.312887+0000","last_fullsized":"2024-03-26T21:31:16.312887+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2823,"log_dups_size":0,"ondisk_log_size":2823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:41.401728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":180672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.174218+0000","last_change":"2024-03-26T21:30:31.741781+0000","last_active":"2024-03-26T21:31:15.174218+0000","last_peered":"2024-03-26T21:31:15.174218+0000","last_clean":"2024-03-26T21:30:19.265083+0000","last_became_active":"2024-03-26T21:30:31.741781+0000","last_became_peered":"2024-03-26T21:30:31.741781+0000","last_unstale":"2024-03-26T21:31:15.174218+0000","last_undegraded":"2024-03-26T21:31:15.174218+0000","last_fullsized":"2024-03-26T21:30:31.735785+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.923329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:25.757267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.174243+0000","last_change":"2024-03-26T21:30:31.743030+0000","last_active":"2024-03-26T21:31:15.174243+0000","last_peered":"2024-03-26T21:31:15.174243+0000","last_clean":"2024-03-26T21:30:19.057039+0000","last_became_active":"2024-03-26T21:30:31.743030+0000","last_became_peered":"2024-03-26T21:30:31.743030+0000","last_unstale":"2024-03-26T21:31:15.174243+0000","last_undegraded":"2024-03-26T21:31:15.174243+0000","last_fullsized":"2024-03-26T21:30:31.730593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.153778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":99,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179082+0000","last_change":"2024-03-26T21:30:31.741495+0000","last_active":"2024-03-26T21:31:15.179082+0000","last_peered":"2024-03-26T21:31:15.179082+0000","last_clean":"2024-03-26T21:30:19.906082+0000","last_became_active":"2024-03-26T21:30:31.741495+0000","last_became_peered":"2024-03-26T21:30:31.741495+0000","last_unstale":"2024-03-26T21:31:15.179082+0000","last_undegraded":"2024-03-26T21:31:15.179082+0000","last_fullsized":"2024-03-26T21:30:31.732617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:17.473710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":78,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.175588+0000","last_change":"2024-03-26T21:30:31.741186+0000","last_active":"2024-03-26T21:31:15.175588+0000","last_peered":"2024-03-26T21:31:15.175588+0000","last_clean":"2024-03-26T21:28:42.246719+0000","last_became_active":"2024-03-26T21:30:31.741186+0000","last_became_peered":"2024-03-26T21:30:31.741186+0000","last_unstale":"2024-03-26T21:31:15.175588+0000","last_undegraded":"2024-03-26T21:30:31.735163+0000","last_fullsized":"2024-03-26T21:30:31.735041+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:30:19.267430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:19.267430+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:21.655140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"105'2733","reported_seq":2763,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:16.316099+0000","last_change":"2024-03-26T21:30:38.551085+0000","last_active":"2024-03-26T21:31:16.316099+0000","last_peered":"2024-03-26T21:31:16.316099+0000","last_clean":"2024-03-26T21:31:16.316099+0000","last_became_active":"2024-03-26T21:30:38.550667+0000","last_became_peered":"2024-03-26T21:30:38.550667+0000","last_unstale":"2024-03-26T21:31:16.316099+0000","last_undegraded":"2024-03-26T21:31:16.316099+0000","last_fullsized":"2024-03-26T21:31:16.316099+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:50.520554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":174912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.172781+0000","last_change":"2024-03-26T21:30:31.739518+0000","last_active":"2024-03-26T21:31:15.172781+0000","last_peered":"2024-03-26T21:31:15.172781+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.739518+0000","last_became_peered":"2024-03-26T21:30:31.739518+0000","last_unstale":"2024-03-26T21:31:15.172781+0000","last_undegraded":"2024-03-26T21:31:15.172781+0000","last_fullsized":"2024-03-26T21:30:31.730680+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:24.244003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T21:31:15.175292+0000","last_change":"2024-03-26T21:30:01.872594+0000","last_active":"2024-03-26T21:31:15.175292+0000","last_peered":"2024-03-26T21:31:15.175292+0000","last_clean":"2024-03-26T21:31:15.175292+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:31:15.175292+0000","last_undegraded":"2024-03-26T21:31:15.175292+0000","last_fullsized":"2024-03-26T21:31:15.175292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:01.872464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.872464+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:16.092905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+undersized","last_fresh":"2024-03-26T21:31:15.179341+0000","last_change":"2024-03-26T21:30:31.743458+0000","last_active":"2024-03-26T21:31:15.179341+0000","last_peered":"2024-03-26T21:31:15.179341+0000","last_clean":"2024-03-26T21:30:19.791720+0000","last_became_active":"2024-03-26T21:30:31.743458+0000","last_became_peered":"2024-03-26T21:30:31.743458+0000","last_unstale":"2024-03-26T21:31:15.179341+0000","last_undegraded":"2024-03-26T21:31:15.179341+0000","last_fullsized":"2024-03-26T21:30:31.731249+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":96,"reported_epoch":105,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:15.179785+0000","last_change":"2024-03-26T21:30:31.741371+0000","last_active":"2024-03-26T21:31:15.179785+0000","last_peered":"2024-03-26T21:31:15.179785+0000","last_clean":"2024-03-26T21:29:37.643762+0000","last_became_active":"2024-03-26T21:30:31.741371+0000","last_became_peered":"2024-03-26T21:30:31.741371+0000","last_unstale":"2024-03-26T21:31:15.179785+0000","last_undegraded":"2024-03-26T21:30:31.731807+0000","last_fullsized":"2024-03-26T21:30:31.731688+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:31:19.853350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2935554048,"num_objects":44793,"num_object_clones":0,"num_object_copies":134379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44793,"num_write_kb":2866752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7886602240,"data_stored":7886602240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44793,"ondisk_log_size":44793,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":150,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":417,"ondisk_log_size":417,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":81,"acting":81,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070501,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905100,"kb_used_data":873924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92840052,"statfs":{"total":95995035648,"available":95068213248,"internally_reserved":0,"allocated":894898176,"data_stored":894485152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364843,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895732,"kb_used_data":864548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92849420,"statfs":{"total":95995035648,"available":95077806080,"internally_reserved":0,"allocated":885297152,"data_stored":884914484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123932,"kb_used_data":1066388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92621220,"statfs":{"total":95995035648,"available":94844129280,"internally_reserved":0,"allocated":1091981312,"data_stored":1091550811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463116,"kb_used_data":1413892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92282036,"statfs":{"total":95995035648,"available":94496804864,"internally_reserved":0,"allocated":1447825408,"data_stored":1447411690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125416,"kb_used_data":1072028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92619736,"statfs":{"total":95995035648,"available":94842609664,"internally_reserved":0,"allocated":1097756672,"data_stored":1097381767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1646672,"kb_used_data":1579400,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92098480,"statfs":{"total":95995035648,"available":94308843520,"internally_reserved":0,"allocated":1617305600,"data_stored":1616886428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27632,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717520,"statfs":{"total":95995035648,"available":95966740480,"internally_reserved":0,"allocated":630784,"data_stored":285906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659182,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732928,"kb_used_data":693428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93012224,"statfs":{"total":95995035648,"available":95244517376,"internally_reserved":0,"allocated":710070272,"data_stored":709671360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1479671808,"data_stored":1479671808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1097990144,"data_stored":1097990144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1107230720,"data_stored":1107230720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1645477888,"data_stored":1645477888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":730464256,"data_stored":730464256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":911343616,"data_stored":911343616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":914423808,"data_stored":914423808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:31:22.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-26T21:31:22.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 29 16 54423 54407 117.23 95.25 0.013113 0.00852342 2024-03-26T21:31:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:22 smithi067 ceph-mon[33346]: pgmap v90: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 2.8 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 717 B/s rd, 85 MiB/s wr, 1.36k op/s; 14148/139218 objects degraded (10.162%) 2024-03-26T21:31:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 14148/134973 objects degraded (10.482%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:22 smithi067 ceph-mon[33346]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T21:31:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:22 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:31:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:22 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:31:22.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:22.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:22.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:22.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:22.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:23.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:23.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:23.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:23.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:23.442 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:31:23.442 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:31:23.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:31:23.442 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:26:15.867860+0000", 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 307.57713281700001, 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.44790030800000002, 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14158", 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/966571165", 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 1 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.443 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867860+0000", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867860+0000", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867861+0000", 2024-03-26T21:31:23.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.141e-06 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867876+0000", 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4708e-05 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867888+0000", 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2067e-05 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867899+0000", 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0981e-05 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867964+0000", 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4820999999999995e-05 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:15.867974+0000", 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.008e-05 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.314206+0000", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.44623207500000001 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.314427+0000", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022053900000000001 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,2", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.314638+0000", 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021143099999999999 2024-03-26T21:31:23.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.315467+0000", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00082871900000000001 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.315647+0000", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018066800000000001 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.315679+0000", 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1325999999999998e-05 2024-03-26T21:31:23.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.315719+0000", 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0673000000000002e-05 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:26:16.315761+0000", 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1078999999999998e-05 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:857 4.4 4:258fd845:::obj_delete_at_hint.0000000113:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:28:42.222158+0000", 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 161.222834941, 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020335789999999999, 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 857 2024-03-26T21:31:23.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222158+0000", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222158+0000", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222159+0000", 2024-03-26T21:31:23.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3799999999999997e-07 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222162+0000", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2210000000000002e-06 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222171+0000", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0359999999999995e-06 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222175+0000", 2024-03-26T21:31:23.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4519999999999999e-06 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222213+0000", 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7330999999999999e-05 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222261+0000", 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8118000000000002e-05 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222360+0000", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8678999999999995e-05 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242380+0000", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020020935 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242413+0000", 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2610000000000001e-05 2024-03-26T21:31:23.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242444+0000", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0474e-05 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242472+0000", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.8248000000000001e-05 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242494+0000", 2024-03-26T21:31:23.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2147999999999999e-05 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.14454.0:857 4.4 e74/68 ondisk, result = 0)", 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:28:42.222842+0000", 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 161.222151349, 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019544517000000001, 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222842+0000", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222842+0000", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222842+0000", 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3499999999999996e-07 2024-03-26T21:31:23.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222846+0000", 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4410000000000002e-06 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242194+0000", 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019348557999999998 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242219+0000", 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4436000000000001e-05 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242372+0000", 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015365999999999999 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242377+0000", 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6820000000000002e-06 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.459 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242386+0000", 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2050000000000008e-06 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.14454.0:857 4.4 e74/68 ondisk, result = 0)", 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:28:42.222860+0000", 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 161.222132977, 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019557641000000001, 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.460 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222860+0000", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222860+0000", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222861+0000", 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6300000000000005e-07 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.222864+0000", 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5609999999999999e-06 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242202+0000", 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019337640999999999 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242226+0000", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4289000000000001e-05 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242406+0000", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000179414 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242410+0000", 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.296e-06 2024-03-26T21:31:23.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:28:42.242418+0000", 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8769999999999993e-06 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:8144 7.b 7:d4fecbe2:::benchmark_data_smithi055_81491_object8143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:30:56.323763+0000", 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.121229846999999, 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.036200403999999999, 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.464 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 8144 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323763+0000", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323763+0000", 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323765+0000", 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.556e-06 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323790+0000", 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.525e-05 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.466 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323809+0000", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8620000000000001e-05 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.323818+0000", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.986e-06 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.324328+0000", 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00051073799999999995 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.324440+0000", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111582 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.324555+0000", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000114553 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.325564+0000", 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0010094240000000001 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359275+0000", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.033710872000000003 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359603+0000", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032793900000000002 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359767+0000", 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000164196 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.470 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359964+0000", 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019668800000000001 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:8166 7.b 7:d87485dc:::benchmark_data_smithi055_81491_object8165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-26T21:31:23.471 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:30:56.340056+0000", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.104937048, 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.021824778999999999, 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 8166 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340056+0000", 2024-03-26T21:31:23.472 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340056+0000", 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340187+0000", 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000131347 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340211+0000", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4083999999999998e-05 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340224+0000", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2164000000000001e-05 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340231+0000", 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6550000000000004e-06 2024-03-26T21:31:23.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340298+0000", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6444000000000003e-05 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340445+0000", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014761400000000001 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.340597+0000", 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000151655 2024-03-26T21:31:23.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.341255+0000", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00065773899999999996 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359320+0000", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.018065392 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.361657+0000", 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0023368080000000001 2024-03-26T21:31:23.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.361831+0000", 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017456899999999999 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.361881+0000", 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9308000000000003e-05 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:8170 7.b 7:d7b0da25:::benchmark_data_smithi055_81491_object8169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:30:56.359838+0000", 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.085155404000002, 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.018930912000000001, 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 8170 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359838+0000", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359838+0000", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359839+0000", 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5540000000000001e-06 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.479 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359870+0000", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1250000000000001e-05 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359881+0000", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0203999999999999e-05 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.359889+0000", 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8990000000000001e-06 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.360037+0000", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014797899999999999 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.360174+0000", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000137527 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.360301+0000", 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012693700000000001 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.361371+0000", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0010696359999999999 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.361998+0000", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00062708700000000002 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.378644+0000", 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016646551999999998 2024-03-26T21:31:23.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.378709+0000", 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4638e-05 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:56.378769+0000", 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9648999999999998e-05 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.483 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:47072 7.b 7:d25f7dca:::benchmark_data_smithi055_81491_object47071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:17.780143+0000", 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.6648498209999998, 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017491422, 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 47072 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.484 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780143+0000", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780143+0000", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780144+0000", 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6199999999999997e-07 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780176+0000", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2283000000000002e-05 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780185+0000", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4249999999999998e-06 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.780192+0000", 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.559e-06 2024-03-26T21:31:23.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.788538+0000", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0083454180000000003 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.788686+0000", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014801199999999999 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.788832+0000", 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000146601 2024-03-26T21:31:23.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.793486+0000", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0046534430000000002 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.793570+0000", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4518999999999995e-05 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.797457+0000", 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038871589999999998 2024-03-26T21:31:23.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.797577+0000", 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119356 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.797635+0000", 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7985000000000001e-05 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.489 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:47076 7.b 7:d53ffc27:::benchmark_data_smithi055_81491_object47075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:17.784713+0000", 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.6602798199999995, 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017829345999999999, 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 47076 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784713+0000", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784713+0000", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784715+0000", 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.73e-06 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784744+0000", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9173999999999999e-05 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784762+0000", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7975999999999999e-05 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.784773+0000", 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0924000000000001e-05 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.793621+0000", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0088481340000000006 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.793784+0000", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000162767 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.793957+0000", 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000173449 2024-03-26T21:31:23.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.799862+0000", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0059041670000000001 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.799908+0000", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6248999999999997e-05 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.802439+0000", 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0025315379999999998 2024-03-26T21:31:23.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.802496+0000", 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7058000000000002e-05 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.802543+0000", 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6180000000000002e-05 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:47081 7.b 7:df1d4cc2:::benchmark_data_smithi055_81491_object47080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:17.785330+0000", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.6596629939999996, 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.01956049, 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 47081 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.496 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785330+0000", 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785330+0000", 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785331+0000", 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8700000000000004e-07 2024-03-26T21:31:23.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785354+0000", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2608000000000001e-05 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785369+0000", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5826999999999999e-05 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.785379+0000", 2024-03-26T21:31:23.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.115e-06 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.797672+0000", 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012293439 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.797883+0000", 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021114500000000001 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.798030+0000", 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014735099999999999 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.802582+0000", 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0045515429999999999 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.802800+0000", 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021839400000000001 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.804734+0000", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001934021 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.804841+0000", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000106774 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:17.804891+0000", 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9385999999999998e-05 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.501 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:48617 7.b 7:d7713012:::benchmark_data_smithi055_81491_object48616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:18.804987+0000", 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 4.6400055919999996, 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.018240287000000001, 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.502 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 48617 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.804987+0000", 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.804987+0000", 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.804988+0000", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9200000000000001e-07 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.805171+0000", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018337900000000001 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.805181+0000", 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4380000000000001e-06 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.805192+0000", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1331999999999999e-05 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.812984+0000", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0077915500000000004 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.813095+0000", 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111369 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.813251+0000", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015551200000000001 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.816996+0000", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037450500000000002 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.819280+0000", 2024-03-26T21:31:23.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0022847789999999998 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.822851+0000", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003570461 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.823176+0000", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032472499999999999 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:18.823228+0000", 2024-03-26T21:31:23.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2200000000000002e-05 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:51780 7.b 7:d6a311c3:::benchmark_data_smithi055_81491_object51779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:20.908141+0000", 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 2.5368519320000003, 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.018662633000000001, 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 51780 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908141+0000", 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908141+0000", 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908143+0000", 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6959999999999999e-06 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908174+0000", 2024-03-26T21:31:23.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1182000000000001e-05 2024-03-26T21:31:23.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:23 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:31:23.511 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:23 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:31:23.511 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:23 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:31:23.511 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:23 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908190+0000", 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6201e-05 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908200+0000", 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6059999999999992e-06 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.908925+0000", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00072503699999999999 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.909044+0000", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119616 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.909272+0000", 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022714499999999999 2024-03-26T21:31:23.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.914395+0000", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0051238639999999997 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.914441+0000", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5201000000000003e-05 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.926711+0000", 2024-03-26T21:31:23.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012270245000000001 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.926761+0000", 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0207999999999998e-05 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.926804+0000", 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2632000000000001e-05 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:53080 7.9 e107/95 7:9d6f926e:::benchmark_data_smithi055_81491_object53079:head v 107'3380, mlcod=107'3376)", 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:21.832761+0000", 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.612231717, 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019444693999999998, 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832761+0000", 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832761+0000", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832762+0000", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3500000000000002e-07 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832783+0000", 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1226999999999999e-05 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832789+0000", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0399999999999998e-06 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.832794+0000", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1059999999999999e-06 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.835104+0000", 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002309714 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.835125+0000", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1526e-05 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.852154+0000", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017028412 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.852206+0000", 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2333999999999997e-05 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.519 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:53486 7.b 7:d5af175c:::benchmark_data_smithi055_81491_object53485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:22.099141+0000", 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.3458520119999999, 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020600618000000001, 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 53486 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.520 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099141+0000", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099141+0000", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099142+0000", 2024-03-26T21:31:23.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4000000000000002e-07 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099175+0000", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3073e-05 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099182+0000", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4320000000000002e-06 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.099188+0000", 2024-03-26T21:31:23.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2099999999999998e-06 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.105064+0000", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058756800000000003 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.105561+0000", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00049713299999999997 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.106027+0000", 2024-03-26T21:31:23.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00046608199999999999 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.111158+0000", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0051308409999999997 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.111198+0000", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0043999999999998e-05 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119472+0000", 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0082736279999999999 2024-03-26T21:31:23.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119694+0000", 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022206999999999999 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119742+0000", 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7885e-05 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.525 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:53489 7.b 7:de06bed8:::benchmark_data_smithi055_81491_object53488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:22.102436+0000", 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.3425566199999999, 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017482311, 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 53489 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102436+0000", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102436+0000", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102438+0000", 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3850000000000001e-06 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.527 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102457+0000", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9338e-05 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102468+0000", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0889e-05 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.102475+0000", 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4039999999999998e-06 2024-03-26T21:31:23.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.111226+0000", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087506679999999996 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.111347+0000", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121165 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.111475+0000", 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012763 2024-03-26T21:31:23.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.117111+0000", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056365590000000002 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.117162+0000", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0924999999999998e-05 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119752+0000", 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0025898679999999999 2024-03-26T21:31:23.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119877+0000", 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012483899999999999 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.119919+0000", 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1641000000000003e-05 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:54438 7.b 7:d3cbfdba:::benchmark_data_smithi055_81491_object54437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:22.720524+0000", 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.72446893099999998, 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017331097, 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.533 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 54438 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720524+0000", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.534 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720524+0000", 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720525+0000", 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8100000000000002e-07 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720545+0000", 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9777000000000001e-05 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720556+0000", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1793999999999999e-05 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.720564+0000", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.306e-06 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.731300+0000", 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010736516999999999 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.731421+0000", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120893 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.731557+0000", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013607199999999999 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.737701+0000", 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0061442149999999997 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.537 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.737713+0000", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1871e-05 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.737745+0000", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1532000000000002e-05 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.737802+0000", 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6776e-05 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.538 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.737855+0000", 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3563000000000002e-05 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.539 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.540 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:54470 7.b 7:db8ec2dc:::benchmark_data_smithi055_81491_object54469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.540 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:22.749743+0000", 2024-03-26T21:31:23.540 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.69524958299999995, 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019904293, 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 54470 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749743+0000", 2024-03-26T21:31:23.541 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749743+0000", 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749744+0000", 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0999999999999998e-07 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.542 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749767+0000", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3283000000000001e-05 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749776+0000", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2190000000000004e-06 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.749782+0000", 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7309999999999996e-06 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.543 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.753827+0000", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040448999999999997 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.753943+0000", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115957 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.754230+0000", 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00028649699999999998 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.544 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.757270+0000", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030401410000000001 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.757316+0000", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5942000000000002e-05 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.769335+0000", 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012018878 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.545 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.769517+0000", 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000182626 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:22.769648+0000", 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013050900000000001 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:55266 7.b 7:dd8331d8:::benchmark_data_smithi055_81491_object55265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.546 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:23.333267+0000", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.111726123, 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.022175707999999999, 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 55266 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333267+0000", 2024-03-26T21:31:23.547 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333267+0000", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333268+0000", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4720000000000001e-06 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333294+0000", 2024-03-26T21:31:23.548 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.5709999999999999e-05 2024-03-26T21:31:23.550 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.550 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.550 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.550 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333307+0000", 2024-03-26T21:31:23.550 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.322e-05 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333316+0000", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6670000000000002e-06 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333662+0000", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00034600899999999999 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333782+0000", 2024-03-26T21:31:23.551 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119968 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333912+0000", 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000130448 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.338656+0000", 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0047440319999999996 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.552 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.338701+0000", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4842999999999997e-05 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355298+0000", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016596722000000001 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355363+0000", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4710999999999998e-05 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.553 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355443+0000", 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9906000000000004e-05 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:55268 7.b 7:d1413543:::benchmark_data_smithi055_81491_object55267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:23.333342+0000", 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.11165069499999999, 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.022211429000000001, 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.554 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 55268 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333342+0000", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333342+0000", 2024-03-26T21:31:23.555 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333343+0000", 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.37e-07 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.556 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333365+0000", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1854000000000001e-05 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333373+0000", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3140000000000004e-06 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.333380+0000", 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1439999999999997e-06 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.557 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.336305+0000", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029242819999999998 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.336454+0000", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000149393 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,6", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.336605+0000", 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015137399999999999 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.558 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.339240+0000", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0026341540000000001 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.339276+0000", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6124999999999997e-05 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355452+0000", 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016176801000000001 2024-03-26T21:31:23.559 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355519+0000", 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6523e-05 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.355554+0000", 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4827999999999999e-05 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.560 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:55271 7.9 e107/95 7:98267f6f:::benchmark_data_smithi055_81491_object55270:head v 107'3532, mlcod=107'3530)", 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:23.335695+0000", 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.10929829200000001, 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016626332000000001, 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335695+0000", 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.561 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335695+0000", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335696+0000", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3399999999999997e-07 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335720+0000", 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4649999999999999e-05 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.562 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335734+0000", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4066999999999999e-05 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335743+0000", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8400000000000001e-06 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335801+0000", 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8028000000000001e-05 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.563 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.335823+0000", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1971e-05 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.352210+0000", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016386364 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:23.352321+0000", 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011147800000000001 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.564 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.565 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.565 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:23.565 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:23.565 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:31:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:23.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:23.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:23.704 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 30 16 55763 55747 116.112 83.75 0.00492067 0.00860745 2024-03-26T21:31:23.705 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:23.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:23 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:31:23.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:23 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:31:23.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:23.835 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:23.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:31:23.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:23.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:23.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:24.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:24.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:24.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:24.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:24.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.422 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:24.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:24.563 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:31:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:31:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[40587]: pgmap v92: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:31:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:31:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:31:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:31:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[29406]: pgmap v92: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:31:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:24 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:31:24.643 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:31:24.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:24.705 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 31 16 57200 57184 115.261 89.8125 0.00798861 0.00866985 2024-03-26T21:31:24.727 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:24.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:24 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:31:24.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:24 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:31:24.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:24 smithi067 ceph-mon[33346]: pgmap v92: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:24.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:24 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:31:24.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:24 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:31:24.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:24.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:24.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:24.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:25.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:25.173 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:25.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:25.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:25.360 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:25.407 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:25.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:25.539 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:31:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:31:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:25 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:31:25.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:25.673 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:25.687 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:25.706 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 32 16 58550 58534 114.292 84.375 0.0200064 0.00874219 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:31:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:25 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:31:25.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:25.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:25.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:25.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:25.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:26.047 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:26.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:26.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:26.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:26.364 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:26.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:26.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:26.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:26.706 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 33 16 59916 59900 113.416 85.375 0.00372265 0.00881117 2024-03-26T21:31:26.714 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:31:26.715 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:31:26.715 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[40587]: pgmap v94: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 940 op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:26.715 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:31:26.715 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:31:26.715 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:26 smithi055 ceph-mon[29406]: pgmap v94: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 940 op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:26.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:26.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:26 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:31:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:26 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:31:26.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:26 smithi067 ceph-mon[33346]: pgmap v94: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 940 op/s; 16460/157542 objects degraded (10.448%) 2024-03-26T21:31:26.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:26.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:26.926 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:26.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:27.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:27.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:27.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:27.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:27.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:27.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:27.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:27.554 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:31:27.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:27.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 16460/157542 objects degraded (10.448%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T21:31:27.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 16460/157542 objects degraded (10.448%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T21:31:27.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:27.706 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 34 16 61262 61246 112.554 84.125 0.00532046 0.00887918 2024-03-26T21:31:27.714 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 16460/157542 objects degraded (10.448%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T21:31:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.761 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:27.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:27.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:27.890 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:27.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:27.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:28.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:28.149 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:28.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:28.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:28.421 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:28.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:28.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:28.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:28.706 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 35 16 62635 62619 111.79 85.8125 0.00792252 0.00893963 2024-03-26T21:31:28.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:31:28.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:31:28.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: pgmap v96: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:28.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: osdmap e110: 8 total, 7 up, 8 in 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:31:28.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:28 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:31:28.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:28.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: pgmap v96: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: osdmap e110: 8 total, 7 up, 8 in 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:31:28.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: pgmap v96: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2810873862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: osdmap e110: 8 total, 7 up, 8 in 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:31:28.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:31:28.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:28.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:28.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:28.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:29.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:29.111 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:29.213 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:29.256 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:29.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:29.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:29.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:29.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.532 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:29.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:29.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:29.706 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 36 15 63886 63871 110.858 78.25 0.0231524 0.00901595 2024-03-26T21:31:29.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:29 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:31:29.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:29 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:31:29.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:29 smithi067 ceph-mon[33346]: 4.a deep-scrub starts 2024-03-26T21:31:29.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:29 smithi067 ceph-mon[33346]: 4.a deep-scrub ok 2024-03-26T21:31:29.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:29.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:29.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:31:29.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:31:29.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[29406]: 4.a deep-scrub starts 2024-03-26T21:31:29.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[29406]: 4.a deep-scrub ok 2024-03-26T21:31:29.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:31:29.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:31:29.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[40587]: 4.a deep-scrub starts 2024-03-26T21:31:29.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:29 smithi055 ceph-mon[40587]: 4.a deep-scrub ok 2024-03-26T21:31:29.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:29.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:29.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:30.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.087 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:30.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:30.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:30.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:30.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:30.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:30.371 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:30.473 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:30.514 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:30.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:30.690 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 5.1e deep-scrub starts 2024-03-26T21:31:30.690 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 5.1e deep-scrub ok 2024-03-26T21:31:30.690 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:31:30.690 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: pgmap v98: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.19k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:31:30.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:31:30.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 5.1e deep-scrub starts 2024-03-26T21:31:30.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 5.1e deep-scrub ok 2024-03-26T21:31:30.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: pgmap v98: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.19k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:31:30.693 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:31:30.694 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:31:30.696 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:30 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:31:30.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 37 16 65253 65237 110.168 85.375 0.0178686 0.00907084 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 5.1e deep-scrub starts 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 5.1e deep-scrub ok 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:31:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: pgmap v98: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.19k op/s; 18656/178896 objects degraded (10.428%) 2024-03-26T21:31:30.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:31:30.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:31:30.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:31:30.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:30 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:31:30.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:30.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:30.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:30.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:30.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:31.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:31.055 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:31.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:31.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:31.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:31.430 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:31.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:31.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:31.630 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:31.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 38 16 66618 66602 109.513 85.3125 0.0092595 0.00912573 2024-03-26T21:31:31.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 3.8 deep-scrub starts 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 3.8 deep-scrub ok 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:31:31.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 3.8 deep-scrub starts 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 3.8 deep-scrub ok 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:31:31.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:31 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:31:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.783 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:31.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:31.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:31.928 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:31.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 3.8 deep-scrub starts 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 3.8 deep-scrub ok 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:31.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:31:31.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:31 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:31:32.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.077 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:32.090 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:31:32.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:32.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:32.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:32.551 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:31:32.559 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:32.587 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:31:32.638 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:31:32.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:32.680 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: pgmap v99: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 18656/185142 objects degraded (10.077%) 2024-03-26T21:31:32.680 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 18656/178896 objects degraded (10.428%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: osdmap e111: 8 total, 7 up, 8 in 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/617010252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: pgmap v99: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 18656/185142 objects degraded (10.077%) 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 18656/178896 objects degraded (10.428%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: osdmap e111: 8 total, 7 up, 8 in 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:31:32.681 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/617010252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:32.682 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:31:32.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 39 16 67712 67696 108.459 68.375 0.0146269 0.00921499 2024-03-26T21:31:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:32.731 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:31:32.777 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:31:32.809 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:32.827 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:31:32.827 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:31:32.827 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:31:32.828 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:31:32.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:32.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: pgmap v99: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 3.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 18656/185142 objects degraded (10.077%) 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 18656/178896 objects degraded (10.428%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: osdmap e111: 8 total, 7 up, 8 in 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:31:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/617010252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:32.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:32.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:33.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:33.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:33.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:33.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:33.541 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 1 pgp_num to 1 2024-03-26T21:31:33.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:33.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:31:33.715470+0000 min lat: 0.00119611 max lat: 0.190367 avg lat: 0.00928342 2024-03-26T21:31:33.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:31:33.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 40 16 68934 68918 107.656 76.375 0.00962622 0.00928342 2024-03-26T21:31:33.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.750 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:33.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:31:33.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:31:33.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:31:33.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:31:33.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:31:33.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:33 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:31:33.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:33.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:33.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:31:33.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:31:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:31:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:33 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:31:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:33 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:31:33.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:34.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:34.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:34.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:34.280 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:34.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:34.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:34.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:34.571 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:34.584 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:31:34.584 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:31:34.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:34.701 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:34.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 41 16 70169 70153 106.913 77.1875 0.00476387 0.00934807 2024-03-26T21:31:34.777 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:34.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: pgmap v101: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 755 B/s rd, 62 MiB/s wr, 984 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:34.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[29406]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: pgmap v101: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 755 B/s rd, 62 MiB/s wr, 984 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:31:34.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:34 smithi055 ceph-mon[40587]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T21:31:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:34.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:34.975 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:34.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: pgmap v101: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 755 B/s rd, 62 MiB/s wr, 984 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2737206902' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:31:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:34 smithi067 ceph-mon[33346]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T21:31:35.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:35.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:35.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:35.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:35.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:35.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:35.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:35.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 42 16 71290 71274 106.035 70.0625 0.0119777 0.00942449 2024-03-26T21:31:35.712 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:35.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:35.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:35 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:31:35.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:35 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:31:35.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:35 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:31:35.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:35 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:31:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:35.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:35.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:35.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:35.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:31:35.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:35 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:31:35.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:35 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:31:36.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:36.115 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:36.217 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:36.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:36.503 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:36.549 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:36.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:36.707 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 43 16 72458 72442 105.267 73 0.00706613 0.00949405 2024-03-26T21:31:36.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:36.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: 6.c deep-scrub starts 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: 6.c deep-scrub ok 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: pgmap v103: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 861 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 20700/198915 objects degraded (10.406%), 27 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[29406]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T21:31:36.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: 6.c deep-scrub starts 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: 6.c deep-scrub ok 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: pgmap v103: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 861 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 20700/198915 objects degraded (10.406%), 27 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:36.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:36 smithi055 ceph-mon[40587]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T21:31:36.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:36.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:36.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:36.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:36.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: 6.c deep-scrub starts 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: 6.c deep-scrub ok 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: pgmap v103: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 861 op/s; 20700/198915 objects degraded (10.406%) 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:31:36.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 20700/198915 objects degraded (10.406%), 27 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:36.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:36 smithi067 ceph-mon[33346]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T21:31:37.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:37.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:37.227 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:37.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:37.459 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.501 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:37.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:37.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:37.631 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:37.708 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 44 15 73638 73623 104.552 73.8125 0.0190681 0.00955989 2024-03-26T21:31:37.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:37.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[40587]: 6.5 deep-scrub starts 2024-03-26T21:31:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[40587]: 6.5 deep-scrub ok 2024-03-26T21:31:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[29406]: 6.5 deep-scrub starts 2024-03-26T21:31:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:37 smithi055 ceph-mon[29406]: 6.5 deep-scrub ok 2024-03-26T21:31:37.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:37.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:37 smithi067 ceph-mon[33346]: 6.5 deep-scrub starts 2024-03-26T21:31:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:37 smithi067 ceph-mon[33346]: 6.5 deep-scrub ok 2024-03-26T21:31:37.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:38.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:38.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:38.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:38.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:38.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:38.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:38.568 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:38.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:38.708 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 45 16 74904 74888 103.986 79.0625 0.0361501 0.00961054 2024-03-26T21:31:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:38.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:38.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[29406]: pgmap v105: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.75k op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:31:38.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:31:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[40587]: pgmap v105: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.75k op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:31:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:31:38.852 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:38.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:38 smithi067 ceph-mon[33346]: pgmap v105: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.75k op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:38 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:31:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:38 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:31:38.993 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:38.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:39.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:39.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:39.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:39.278 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:39.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:31:39.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:39.401 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:39.433 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:39.535 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.585 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:39.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:39.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:39.708 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 46 16 76088 76072 103.334 74 0.0101868 0.00967125 2024-03-26T21:31:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.730 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:39.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:39.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:39.857 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[29406]: 6.9 deep-scrub starts 2024-03-26T21:31:39.857 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[29406]: 6.9 deep-scrub ok 2024-03-26T21:31:39.857 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:31:39.857 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:31:39.858 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[40587]: 6.9 deep-scrub starts 2024-03-26T21:31:39.858 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[40587]: 6.9 deep-scrub ok 2024-03-26T21:31:39.858 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:31:39.858 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:39 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:31:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:39.878 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:39.890 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:39 smithi067 ceph-mon[33346]: 6.9 deep-scrub starts 2024-03-26T21:31:39.890 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:39 smithi067 ceph-mon[33346]: 6.9 deep-scrub ok 2024-03-26T21:31:39.890 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:39 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:31:39.890 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:39 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:31:39.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:40.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:40.142 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.196 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:40.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:40.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:40.324 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:40.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:40.453 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:40.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:40.598 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:40.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:40.708 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 47 16 77141 77125 102.535 65.8125 0.00768105 0.00974721 2024-03-26T21:31:40.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 4.16 deep-scrub starts 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 4.16 deep-scrub ok 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: pgmap v106: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 658 B/s rd, 47 MiB/s wr, 760 op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 3.c deep-scrub starts 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[29406]: 3.c deep-scrub ok 2024-03-26T21:31:40.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 4.16 deep-scrub starts 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 4.16 deep-scrub ok 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: pgmap v106: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 658 B/s rd, 47 MiB/s wr, 760 op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 3.c deep-scrub starts 2024-03-26T21:31:40.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:40 smithi055 ceph-mon[40587]: 3.c deep-scrub ok 2024-03-26T21:31:40.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:40.871 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:40.872 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:40.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:40.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 4.16 deep-scrub starts 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 4.16 deep-scrub ok 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: pgmap v106: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 658 B/s rd, 47 MiB/s wr, 760 op/s; 22555/216642 objects degraded (10.411%) 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 3.c deep-scrub starts 2024-03-26T21:31:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:40 smithi067 ceph-mon[33346]: 3.c deep-scrub ok 2024-03-26T21:31:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:41.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:41.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:41.128 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:41.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:41.277 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:41.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:41.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:41.508 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:41.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:41.655 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:41.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:41.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 48 16 78128 78112 101.683 61.6875 0.0159716 0.00982948 2024-03-26T21:31:41.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:41.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:41.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:41.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:41 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:31:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:41 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:31:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:41 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:31:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:41 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:31:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:41 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 22555/216642 objects degraded (10.411%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:42.000 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:42.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 22555/216642 objects degraded (10.411%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:31:42.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:41 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 22555/216642 objects degraded (10.411%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:42.102 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:42.140 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:42.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:42.290 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:42.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:42.456 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:42.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:42.585 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:42.659 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:31:42.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:42.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 49 15 79224 79209 101.007 68.5625 0.00691262 0.00989514 2024-03-26T21:31:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.727 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:42.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:42.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:42.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T21:31:42.872 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:31:42.872 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T21:31:42.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T21:31:42.918 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: pgmap v107: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s; 22555/222207 objects degraded (10.150%) 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/79995101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: pgmap v107: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s; 22555/222207 objects degraded (10.150%) 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/79995101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.919 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:42 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: pgmap v107: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s; 22555/222207 objects degraded (10.150%) 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/79995101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:42 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:31:42.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:42.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:43.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:43.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:43.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:43.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:43.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:43.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:43.547 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:43.649 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.691 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:43.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:43.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 50 16 80267 80251 100.289 65.125 0.00691898 0.00996619 2024-03-26T21:31:43.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:31:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[40587]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:31:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:43 smithi055 ceph-mon[29406]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T21:31:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.837 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:43.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:43.938 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:43.977 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:43.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:43 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:31:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:43 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:31:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:43 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:43 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:31:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:43 smithi067 ceph-mon[33346]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T21:31:44.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:44.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:44.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.244 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:44.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:44.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:44.383 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:44.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:44.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:44.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:44.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 51 16 81270 81254 99.5517 62.6875 0.00719954 0.0100264 2024-03-26T21:31:44.743 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:44.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:44.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:44.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:44.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:44 smithi067 ceph-mon[33346]: pgmap v110: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 754 B/s rd, 53 MiB/s wr, 855 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:45.024 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:44 smithi055 ceph-mon[40587]: pgmap v110: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 754 B/s rd, 53 MiB/s wr, 855 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:45.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:44 smithi055 ceph-mon[29406]: pgmap v110: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 754 B/s rd, 53 MiB/s wr, 855 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:45.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:45.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.197 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:45.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:45.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:45.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:45.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.504 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:45.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:45.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:45.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 52 16 82450 82434 99.0555 73.75 0.00825321 0.01009 2024-03-26T21:31:45.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:45.861 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:45.904 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:45.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:46.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.045 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:46.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:46.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:46.173 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:46.275 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:46.334 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:46.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:46.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:46.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.630 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:46.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:46.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[40587]: pgmap v111: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:46.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:46.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:46.676 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 24407/234051 objects degraded (10.428%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:46.676 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[29406]: pgmap v111: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:46.676 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:46.676 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:46.676 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:46 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 24407/234051 objects degraded (10.428%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:46.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 53 16 83579 83563 98.5178 70.5625 0.0169749 0.0101447 2024-03-26T21:31:46.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:46.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:46.802 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:31:46.855 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:31:46.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:46.916 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:31:46.921 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:46.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:46.969 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:31:46.992 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:46 smithi067 ceph-mon[33346]: pgmap v111: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 4.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 24407/234051 objects degraded (10.428%) 2024-03-26T21:31:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:31:46.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:31:46.993 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:46 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 24407/234051 objects degraded (10.428%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:47.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:47.025 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:31:47.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:47.074 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:31:47.121 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:31:47.121 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:31:47.121 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:31:47.121 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:31:47.121 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:31:47.122 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:31:47.122 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:31:47.122 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:31:47.122 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:31:47.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:47.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:47.224 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:31:47.224 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:31:47.224 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:31:47.224 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:31:47.224 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:31:47.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:47.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:47.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:47.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:47.612 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:47.709 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 54 16 84584 84568 97.8566 62.8125 0.0105782 0.0102141 2024-03-26T21:31:47.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:47.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:47.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:47.883 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:47.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:47.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:47.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:47.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:47 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2418471168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:48.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:48.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:47 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2418471168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:48.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:47 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2418471168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:31:48.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:31:48.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:48.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:48.394 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:48.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:48.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:48.590 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:48.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:48.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 55 16 85686 85670 97.3296 68.875 0.0163413 0.0102692 2024-03-26T21:31:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:48.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:48.779 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:48 smithi055 ceph-mon[40587]: pgmap v112: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:48.780 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:48 smithi055 ceph-mon[29406]: pgmap v112: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:48.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:31:48.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:48.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:48.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:48.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:48 smithi067 ceph-mon[33346]: pgmap v112: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:48.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:48.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:49.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:49.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:49.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:49.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:49.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:31:49.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:31:49.439 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:49.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:49.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:49.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:49.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 56 16 86806 86790 96.8416 70 0.00812924 0.0103219 2024-03-26T21:31:49.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:49.772 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:49.788 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:49.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:49.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:49.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:50.047 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:50.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:50.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:50.342 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:50.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:50.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:50.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:50.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:50.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 57 16 87922 87906 96.3662 69.75 0.0160117 0.0103722 2024-03-26T21:31:50.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:50.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:50 smithi055 ceph-mon[29406]: pgmap v113: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:50.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:50 smithi055 ceph-mon[40587]: pgmap v113: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:50.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:50.884 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:50.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:50.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:50.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:50 smithi067 ceph-mon[33346]: pgmap v113: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 26113/250251 objects degraded (10.435%) 2024-03-26T21:31:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:51.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:51.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:51.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:51.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:51.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:51.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:51.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:51.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:31:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:51.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:51.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:51.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:51.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 58 16 88999 88983 95.8653 67.3125 0.0140632 0.0104259 2024-03-26T21:31:51.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:51.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:51.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:51 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 26113/250251 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:51.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:51 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 26113/250251 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:51.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:51.875 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:51 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 26113/250251 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:51.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:51.999 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:51.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:52.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:52.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:52.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:52.257 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:52.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:52.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.423 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:52.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:52.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:52.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:52.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:52.683 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:52.704 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:52 smithi055 ceph-mon[40587]: pgmap v114: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 737 op/s; 26113/255447 objects degraded (10.222%) 2024-03-26T21:31:52.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:52 smithi055 ceph-mon[29406]: pgmap v114: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 737 op/s; 26113/255447 objects degraded (10.222%) 2024-03-26T21:31:52.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 59 16 90080 90064 95.3855 67.5625 0.0139999 0.0104778 2024-03-26T21:31:52.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:52.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:52.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:52.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:52.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:52 smithi067 ceph-mon[33346]: pgmap v114: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 737 op/s; 26113/255447 objects degraded (10.222%) 2024-03-26T21:31:53.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:53.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:53.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.230 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:53.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:53.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:53.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.523 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:53.540 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "epoch": 115, 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "created": "2024-03-26T21:22:56.673257+0000", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "modified": "2024-03-26T21:31:42.627708+0000", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "last_up_change": "2024-03-26T21:30:30.724748+0000", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "last_in_change": "2024-03-26T21:28:02.902888+0000", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_num": 5801984, 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_set": [ 2024-03-26T21:31:53.541 INFO:teuthology.orchestra.run.smithi055.stdout: "noscrub", 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "pglog_hardlimit", 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "purged_snapdirs", 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_deletes", 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "sortbitwise" 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_version": 18, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_snapshot": "", 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_max": 7, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "max_osd": 8, 2024-03-26T21:31:53.542 INFO:teuthology.orchestra.run.smithi055.stdout: "require_min_compat_client": "luminous", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "min_compat_client": "luminous", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "require_osd_release": "squid", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "allow_crimson": false, 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "pools": [ 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 1, 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".mgr", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:26:13.884991+0000", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.543 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 1, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 1, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 1, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 1, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 1, 2024-03-26T21:31:53.544 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "112", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.545 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.547 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.548 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.548 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.549 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_max": 32, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_min": 1 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.550 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": {} 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 7.8899998664855957, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 7.8899998664855957, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 3, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 3, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.551 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 2, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "datapool", 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.142617+0000", 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 8193, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.552 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 3, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 3, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 3, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 3, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 3, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.553 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "71", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 2, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 71, 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.554 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.555 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.556 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.556 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.556 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.556 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.557 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.557 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.557 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.557 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.557 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd": {} 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.558 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.6500000953674316, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.6500000953674316, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.559 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 3, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".rgw.root", 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.591599+0000", 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.560 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.561 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "68", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.562 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.563 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.564 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.7400000095367432, 2024-03-26T21:31:53.565 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.7400000095367432, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.566 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.567 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 4, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.log", 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:34.904126+0000", 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.568 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:31:53.569 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "70", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.570 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.571 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.572 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:31:53.573 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.4900000095367432, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.4900000095367432, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.574 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 5, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.control", 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:36.852373+0000", 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.575 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.576 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.577 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.577 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "72", 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.578 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.579 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.580 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.581 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.4900000095367432, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.4900000095367432, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 6, 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.meta", 2024-03-26T21:31:53.582 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:38.937273+0000", 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.583 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "74", 2024-03-26T21:31:53.584 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.585 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.586 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.587 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.588 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.588 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:31:53.588 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_bias": 4 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.9900000095367432, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.9900000095367432, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.589 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 7, 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "unique_pool_0", 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:30:37.189111+0000", 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:31:53.590 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 16, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 16, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 16, 2024-03-26T21:31:53.591 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 16, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 16, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "100", 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:31:53.592 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:31:53.593 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:31:53.594 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "rados": {} 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:31:53.595 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.9900000095367432, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.9900000095367432, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:31:53.596 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae", 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 8, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6801", 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6802", 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6804", 2024-03-26T21:31:53.599 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6803", 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:31:53.600 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6801/317381212", 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6802/317381212", 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6804/317381212", 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6803/317381212", 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.601 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db", 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 13, 2024-03-26T21:31:53.602 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6805", 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.603 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6806", 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6808", 2024-03-26T21:31:53.604 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6807", 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6805/3733554168", 2024-03-26T21:31:53.605 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6806/3733554168", 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6808/3733554168", 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6807/3733554168", 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b", 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.606 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 18, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6809", 2024-03-26T21:31:53.607 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6810", 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.608 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6812", 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.609 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6811", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6809/3059242923", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6810/3059242923", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6812/3059242923", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6811/3059242923", 2024-03-26T21:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b", 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.611 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 26, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.612 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6813", 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6814", 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:31:53.613 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6816", 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.614 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6815", 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:31:53.615 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6813/3851321468", 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6814/3851321468", 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6816/3851321468", 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6815/3851321468", 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.616 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598", 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 0, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 31, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 72, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 93, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.617 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6800", 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1777724250 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6801", 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1777724250 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.618 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6803", 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1777724250 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.619 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6802", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 1777724250 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6800/1777724250", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6801/1777724250", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6803/1777724250", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6802/1777724250", 2024-03-26T21:31:53.620 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: "exists" 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213", 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.621 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 41, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.622 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6804", 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6805", 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.623 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6807", 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6806", 2024-03-26T21:31:53.624 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6804/278711358", 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6805/278711358", 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6807/278711358", 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6806/278711358", 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.625 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9", 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 50, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.626 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6808", 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6809", 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.627 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6811", 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.628 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6810", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6808/3926251639", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6809/3926251639", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6811/3926251639", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6810/3926251639", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.629 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55", 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:31:53.630 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 59, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 95, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6812", 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.631 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6813", 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6815", 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6814", 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6812/2981645260", 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6813/2981645260", 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6815/2981645260", 2024-03-26T21:31:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6814/2981645260", 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_xinfo": [ 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:31:23.852731+0000", 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:30:56.674991+0000", 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:31:14.187629+0000", 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.638 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:31:41.260289+0000", 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:29:36.155451+0000", 2024-03-26T21:31:53.639 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 92 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:30:39.238530+0000", 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.640 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:30:41.238479+0000", 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:31:03.753703+0000", 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap": [], 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_items": [ 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.4", 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.642 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.6", 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.17", 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:31:53.643 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 2 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_primaries": [], 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_temp": [], 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_temp": [], 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklist": { 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2649740776": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/4141219490": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/147833426": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2854772543": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/2804789175": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2199694469": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:31:53.644 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1614234257": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2834138692": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1740219825": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/268664696": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/536717527": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/966571165": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2757111431": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/559658018": "2024-03-27T21:23:57.787077+0000" 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "range_blocklist": {}, 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profiles": { 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "default": { 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "crush-failure-domain": "osd", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "k": "2", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "m": "1", 2024-03-26T21:31:53.645 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin": "jerasure", 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "technique": "reed_sol_van" 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps_queue": [], 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "new_removed_snaps": [], 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "new_purged_snaps": [], 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_node_flags": {}, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "device_class_flags": {}, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode": { 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_enabled": false, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_bucket_count": 0, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "degraded_stretch_mode": 0, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "recovering_stretch_mode": 0, 2024-03-26T21:31:53.646 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_bucket": 0 2024-03-26T21:31:53.647 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:53.647 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:31:53.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.694 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:53.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:31:53.718336+0000 min lat: 0.00119611 max lat: 0.190367 avg lat: 0.0105398 2024-03-26T21:31:53.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:31:53.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 60 16 91072 91056 94.8291 62 0.0123183 0.0105398 2024-03-26T21:31:53.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:53.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:53.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:53.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/627726597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:31:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:53.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:53.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:54.039 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/627726597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:31:54.039 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/627726597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:31:54.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:54.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:54.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:54.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:54.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:54.410 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:54.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:54.564 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.565 INFO:tasks.thrashosds.thrasher:j is {'epoch': 115, 'fsid': 'e5f52b02-ebb6-11ee-95d0-87774f69a715', 'created': '2024-03-26T21:22:56.673257+0000', 'modified': '2024-03-26T21:31:42.627708+0000', 'last_up_change': '2024-03-26T21:30:30.724748+0000', 'last_in_change': '2024-03-26T21:28:02.902888+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T21:28:33.142617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T21:28:33.591599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T21:28:34.904126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T21:28:36.852373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T21:28:38.937273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-26T21:30:37.189111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4f487071-c151-406b-b25b-3135cb3c63ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6801', 'nonce': 317381212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6802', 'nonce': 317381212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6804', 'nonce': 317381212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6803', 'nonce': 317381212}]}, 'public_addr': '172.21.15.55:6801/317381212', 'cluster_addr': '172.21.15.55:6802/317381212', 'heartbeat_back_addr': '172.21.15.55:6804/317381212', 'heartbeat_front_addr': '172.21.15.55:6803/317381212', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6805', 'nonce': 3733554168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6806', 'nonce': 3733554168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6808', 'nonce': 3733554168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6807', 'nonce': 3733554168}]}, 'public_addr': '172.21.15.55:6805/3733554168', 'cluster_addr': '172.21.15.55:6806/3733554168', 'heartbeat_back_addr': '172.21.15.55:6808/3733554168', 'heartbeat_front_addr': '172.21.15.55:6807/3733554168', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6809', 'nonce': 3059242923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6810', 'nonce': 3059242923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6812', 'nonce': 3059242923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6811', 'nonce': 3059242923}]}, 'public_addr': '172.21.15.55:6809/3059242923', 'cluster_addr': '172.21.15.55:6810/3059242923', 'heartbeat_back_addr': '172.21.15.55:6812/3059242923', 'heartbeat_front_addr': '172.21.15.55:6811/3059242923', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '30637a8d-5c1b-4ede-8214-a4595c875d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6813', 'nonce': 3851321468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6814', 'nonce': 3851321468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6816', 'nonce': 3851321468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6815', 'nonce': 3851321468}]}, 'public_addr': '172.21.15.55:6813/3851321468', 'cluster_addr': '172.21.15.55:6814/3851321468', 'heartbeat_back_addr': '172.21.15.55:6816/3851321468', 'heartbeat_front_addr': '172.21.15.55:6815/3851321468', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'cd2d68be-4b10-4242-826b-e085c2ce5598', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 72, 'down_at': 93, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 1777724250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6801', 'nonce': 1777724250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 1777724250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 1777724250}]}, 'public_addr': '172.21.15.67:6800/1777724250', 'cluster_addr': '172.21.15.67:6801/1777724250', 'heartbeat_back_addr': '172.21.15.67:6803/1777724250', 'heartbeat_front_addr': '172.21.15.67:6802/1777724250', 'state': ['exists']}, {'osd': 5, 'uuid': '7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 278711358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6805', 'nonce': 278711358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 278711358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 278711358}]}, 'public_addr': '172.21.15.67:6804/278711358', 'cluster_addr': '172.21.15.67:6805/278711358', 'heartbeat_back_addr': '172.21.15.67:6807/278711358', 'heartbeat_front_addr': '172.21.15.67:6806/278711358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd47c79f3-96bf-46b6-a7f3-a315446d04b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 3926251639}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 3926251639}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 3926251639}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 3926251639}]}, 'public_addr': '172.21.15.67:6808/3926251639', 'cluster_addr': '172.21.15.67:6809/3926251639', 'heartbeat_back_addr': '172.21.15.67:6811/3926251639', 'heartbeat_front_addr': '172.21.15.67:6810/3926251639', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f10b98b8-887d-4f83-8bc9-99f5c302dc55', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 2981645260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6813', 'nonce': 2981645260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 2981645260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2981645260}]}, 'public_addr': '172.21.15.67:6812/2981645260', 'cluster_addr': '172.21.15.67:6813/2981645260', 'heartbeat_back_addr': '172.21.15.67:6815/2981645260', 'heartbeat_front_addr': '172.21.15.67:6814/2981645260', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:31:23.852731+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:30:56.674991+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:31:14.187629+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:31:41.260289+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:29:36.155451+0000', 'dead_epoch': 92}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:30:39.238530+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:30:41.238479+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:31:03.753703+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.55:0/2649740776': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/4141219490': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/147833426': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2854772543': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/2804789175': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/2199694469': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/1614234257': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/2834138692': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/1740219825': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/268664696': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:6800/536717527': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/966571165': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2757111431': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/559658018': '2024-03-27T21:23:57.787077+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T21:31:54.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:31:54.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:54.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:54.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 61 16 92166 92150 94.3953 68.375 0.00903127 0.0105873 2024-03-26T21:31:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:54.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:54.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:54 smithi055 ceph-mon[29406]: pgmap v115: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 63 MiB/s wr, 1.02k op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:54.811 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:54 smithi055 ceph-mon[40587]: pgmap v115: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 63 MiB/s wr, 1.02k op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:54.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:54.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:54.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:54.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:54 smithi067 ceph-mon[33346]: pgmap v115: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 959 B/s rd, 63 MiB/s wr, 1.02k op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:55.031 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:55.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.189 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:55.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:55.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:55.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:55.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:55.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:55.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:55.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:55.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 62 16 93219 93203 93.9343 65.8125 0.00470002 0.0106405 2024-03-26T21:31:55.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:55.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:55.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:55.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:55.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:56.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.061 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:56.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:56.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.198 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:50979 7.e 7:70930ac8:::benchmark_data_smithi055_81491_object50978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:20.286840+0000", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 35.915701028999997, 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.056058510999999998, 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 50979 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286840+0000", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286840+0000", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286842+0000", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.531e-06 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286872+0000", 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0073000000000002e-05 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286884+0000", 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1803e-05 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.286892+0000", 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9980000000000003e-06 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.287408+0000", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00051608299999999997 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.287552+0000", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000144213 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.287710+0000", 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015798 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.291700+0000", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0039901440000000002 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.291715+0000", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4943000000000001e-05 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.342045+0000", 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.050330411999999998 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.342847+0000", 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00080177600000000003 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:20.342899+0000", 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1554999999999997e-05 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.204 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:51932 7.e 7:7c9ae7ce:::benchmark_data_smithi055_81491_object51931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:21.004698+0000", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 35.197843053, 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.055593895999999997, 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 51932 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004698+0000", 2024-03-26T21:31:56.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004698+0000", 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004699+0000", 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7800000000000001e-07 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004721+0000", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1767999999999999e-05 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004728+0000", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5310000000000004e-06 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.004734+0000", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3260000000000001e-06 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.008108+0000", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033736339999999999 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.008232+0000", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012364599999999999 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.008363+0000", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013152500000000001 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.010437+0000", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002073292 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.010980+0000", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00054331400000000004 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.060139+0000", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.049159204999999997 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.060242+0000", 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103309 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.060292+0000", 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9768000000000002e-05 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:51947 7.e 7:7789d2a0:::benchmark_data_smithi055_81491_object51946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:21.013463+0000", 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 35.189078299000002, 2024-03-26T21:31:56.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.055656288999999998, 2024-03-26T21:31:56.211 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.211 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.211 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.211 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.211 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 51947 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013463+0000", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013463+0000", 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013464+0000", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.031e-06 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013483+0000", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8898000000000001e-05 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013491+0000", 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7500000000000003e-06 2024-03-26T21:31:56.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013496+0000", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5509999999999999e-06 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013538+0000", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2029e-05 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013614+0000", 2024-03-26T21:31:56.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5211999999999997e-05 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.013696+0000", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2395e-05 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.015033+0000", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001337466 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.015359+0000", 2024-03-26T21:31:56.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032604299999999998 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.068986+0000", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.053626697000000001 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.069043+0000", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7129000000000001e-05 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:21.069119+0000", 2024-03-26T21:31:56.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6087999999999994e-05 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:62671 7.a e110/95 7:50fd3a7e:::benchmark_data_smithi055_81491_object62670:head v 110'3874, mlcod=110'3868)", 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.751307+0000", 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.451234618000001, 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.065019594, 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.217 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751307+0000", 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751307+0000", 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751307+0000", 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3600000000000003e-07 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751331+0000", 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3388999999999999e-05 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751340+0000", 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0610000000000002e-06 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751347+0000", 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1210000000000001e-06 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751418+0000", 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1094999999999996e-05 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.756451+0000", 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.005033147 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816299+0000", 2024-03-26T21:31:56.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.059847412000000003 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816326+0000", 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7733e-05 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:62678 7.2 e110/95 7:418eb108:::benchmark_data_smithi055_81491_object62677:head v 110'3884, mlcod=110'3881)", 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.751446+0000", 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.451094993000002, 2024-03-26T21:31:56.222 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.064846841000000002, 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751446+0000", 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751446+0000", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751447+0000", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4300000000000003e-07 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751487+0000", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0615999999999999e-05 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751495+0000", 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9300000000000003e-06 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751502+0000", 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2999999999999998e-06 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751564+0000", 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1854999999999996e-05 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.751582+0000", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8661e-05 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816248+0000", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.064665916000000004 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816293+0000", 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5019999999999999e-05 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.226 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:62693 7.e 7:760f9e93:::benchmark_data_smithi055_81491_object62692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.753403+0000", 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.449138153, 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.062921115, 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 62693 2024-03-26T21:31:56.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753403+0000", 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753403+0000", 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753404+0000", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8700000000000004e-07 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753425+0000", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0876000000000001e-05 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753435+0000", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5670000000000001e-06 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.753442+0000", 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5039999999999996e-06 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754369+0000", 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00092744099999999999 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754482+0000", 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011217099999999999 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754630+0000", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014793400000000001 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757063+0000", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024338770000000001 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757084+0000", 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0710999999999999e-05 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816207+0000", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.059123270999999998 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816277+0000", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9581999999999999e-05 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816324+0000", 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7293999999999998e-05 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.232 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:62686 7.2 e110/95 7:43667907:::benchmark_data_smithi055_81491_object62685:head v 110'3885, mlcod=110'3881)", 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.754270+0000", 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.448271254000002, 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.066504675999999999, 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754270+0000", 2024-03-26T21:31:56.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754270+0000", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754271+0000", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.85e-07 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754302+0000", 2024-03-26T21:31:56.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0800000000000003e-05 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754311+0000", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9120000000000004e-06 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.754317+0000", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7989999999999996e-06 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816329+0000", 2024-03-26T21:31:56.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.062011269000000001 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816350+0000", 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1362e-05 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.820717+0000", 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0043669720000000002 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.820775+0000", 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7676999999999997e-05 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:62694 7.5 e110/95 7:aed8b81b:::benchmark_data_smithi055_81491_object62693:head v 110'3956, mlcod=110'3954)", 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.755252+0000", 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.447288932999999, 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.062877997000000005, 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.237 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755252+0000", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755252+0000", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755253+0000", 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7599999999999997e-07 2024-03-26T21:31:56.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755343+0000", 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9864000000000006e-05 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755352+0000", 2024-03-26T21:31:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.240 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:56.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3479999999999993e-06 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755358+0000", 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1299999999999998e-06 2024-03-26T21:31:56.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755971+0000", 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00061272099999999999 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.755991+0000", 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9686000000000001e-05 2024-03-26T21:31:56.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816344+0000", 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.060353192999999999 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.818130+0000", 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0017864790000000001 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.243 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:62696 7.5 e110/95 7:a23e8e9e:::benchmark_data_smithi055_81491_object62695:head v 110'3957, mlcod=110'3954)", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.757311+0000", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.445230511999998, 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.060895657999999998, 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757311+0000", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757311+0000", 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757311+0000", 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6000000000000003e-07 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757384+0000", 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2198000000000002e-05 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757391+0000", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.951e-06 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757397+0000", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4330000000000001e-06 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757623+0000", 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022597200000000001 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.757636+0000", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2588e-05 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.818135+0000", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.060499021 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.818206+0000", 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1835000000000006e-05 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.247 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.24704.0:62693 7.e e110/95 ondisk, result = 0)", 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:28.758161+0000", 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.444380420000002, 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.058171291, 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758161+0000", 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758161+0000", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758162+0000", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3399999999999999e-07 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758166+0000", 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1570000000000002e-06 2024-03-26T21:31:56.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758175+0000", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8610000000000004e-06 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758181+0000", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6630000000000004e-06 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758247+0000", 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5356000000000002e-05 2024-03-26T21:31:56.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.758253+0000", 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9129999999999997e-06 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:28.816332+0000", 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.058078707 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.251 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:63133 7.e 7:74eec2e5:::benchmark_data_smithi055_81491_object63132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:29.129984+0000", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 27.072557103000001, 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.062395842, 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 63133 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.129984+0000", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.129984+0000", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.129985+0000", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0270000000000001e-06 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130035+0000", 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9455e-05 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130044+0000", 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7440000000000002e-06 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130052+0000", 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3409999999999997e-06 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130104+0000", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2139999999999999e-05 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130214+0000", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000110089 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.130296+0000", 2024-03-26T21:31:56.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1793999999999996e-05 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.132033+0000", 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001736949 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.132155+0000", 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000122588 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.192289+0000", 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.060134082999999998 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.192340+0000", 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0961000000000002e-05 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:29.192380+0000", 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9671e-05 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:87818 7.e 7:75b612d2:::benchmark_data_smithi055_81491_object87817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:50.599180+0000", 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.6033608390000005, 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.051161369999999998, 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 87818 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.258 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599180+0000", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599180+0000", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599185+0000", 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.177e-06 2024-03-26T21:31:56.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599206+0000", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1273000000000001e-05 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599217+0000", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0654e-05 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.599224+0000", 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8790000000000003e-06 2024-03-26T21:31:56.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.601646+0000", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002421861 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.601781+0000", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000134284 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.601914+0000", 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000133091 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.606296+0000", 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0043817889999999996 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.606308+0000", 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2238e-05 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.650068+0000", 2024-03-26T21:31:56.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.043760594999999999 2024-03-26T21:31:56.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.650252+0000", 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000183656 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.650342+0000", 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9872999999999998e-05 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:87828 7.e 7:716c30b5:::benchmark_data_smithi055_81491_object87827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:50.608308+0000", 2024-03-26T21:31:56.264 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.5942328579999998, 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.048123613000000003, 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 87828 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608308+0000", 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608308+0000", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608310+0000", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3069999999999999e-06 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608332+0000", 2024-03-26T21:31:56.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1831000000000002e-05 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608342+0000", 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0421000000000001e-05 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608349+0000", 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3379999999999999e-06 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608443+0000", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4077000000000001e-05 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608582+0000", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000138727 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608769+0000", 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018692900000000001 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.269 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.613791+0000", 2024-03-26T21:31:56.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050214309999999998 2024-03-26T21:31:56.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.613801+0000", 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0597e-05 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656182+0000", 2024-03-26T21:31:56.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.042380813000000003 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656281+0000", 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8765999999999996e-05 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656432+0000", 2024-03-26T21:31:56.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000151376 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:87830 7.e 7:7468c785:::benchmark_data_smithi055_81491_object87829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:50.608370+0000", 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 5.5941714119999997, 2024-03-26T21:31:56.273 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.048257566000000002, 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 87830 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608370+0000", 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608370+0000", 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608370+0000", 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3300000000000002e-07 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608390+0000", 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9843999999999998e-05 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608397+0000", 2024-03-26T21:31:56.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7900000000000002e-06 2024-03-26T21:31:56.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.608403+0000", 2024-03-26T21:31:56.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5899999999999998e-06 2024-03-26T21:31:56.278 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.278 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.278 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.278 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.611626+0000", 2024-03-26T21:31:56.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032234849999999999 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.611734+0000", 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010798699999999999 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.611859+0000", 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000125303 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.614042+0000", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0021828300000000002 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.614106+0000", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3231999999999994e-05 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656477+0000", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.042371574000000002 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656558+0000", 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0667000000000005e-05 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:50.656627+0000", 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9731e-05 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.281 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:90073 7.e 7:701f8707:::benchmark_data_smithi055_81491_object90072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:52.707554+0000", 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 3.494987391, 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.050609266999999999, 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 90073 2024-03-26T21:31:56.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707554+0000", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707554+0000", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707555+0000", 2024-03-26T21:31:56.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9699999999999995e-07 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707579+0000", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4128999999999999e-05 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707594+0000", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5415999999999999e-05 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.707604+0000", 2024-03-26T21:31:56.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3530000000000001e-06 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.709936+0000", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0023324169999999998 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.710821+0000", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00088523699999999998 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.710992+0000", 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017030300000000001 2024-03-26T21:31:56.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.718797+0000", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0078056499999999999 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.719273+0000", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000476102 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.758039+0000", 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.038766087999999997 2024-03-26T21:31:56.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.758103+0000", 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3257999999999996e-05 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:52.758163+0000", 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0516999999999997e-05 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:91908 7.e 7:753b30e2:::benchmark_data_smithi055_81491_object91907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:54.457182+0000", 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.7453591400000001, 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.058770189, 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.288 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91908 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457182+0000", 2024-03-26T21:31:56.289 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457182+0000", 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457183+0000", 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.27e-07 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457203+0000", 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0085000000000002e-05 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457208+0000", 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1139999999999997e-06 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.457213+0000", 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8749999999999999e-06 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.481296+0000", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.024082823 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.481415+0000", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119794 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.481552+0000", 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000136227 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.494879+0000", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.013327847 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.494950+0000", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0395999999999995e-05 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.515741+0000", 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020791185 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.515895+0000", 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015452700000000001 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.515952+0000", 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6888999999999997e-05 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:91916 7.e 7:7a53b760:::benchmark_data_smithi055_81491_object91915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.294 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:54.468721+0000", 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.7338206940000001, 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.047340663999999998, 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91916 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468721+0000", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468721+0000", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468722+0000", 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5269999999999999e-06 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468746+0000", 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3799999999999999e-05 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468756+0000", 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7000000000000003e-06 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.468763+0000", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2559999999999996e-06 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.491285+0000", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.02252229 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.491537+0000", 2024-03-26T21:31:56.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025212100000000001 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.491690+0000", 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000152583 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.494996+0000", 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003306186 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504114+0000", 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0091180519999999998 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.515963+0000", 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011848687 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.516003+0000", 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9789000000000001e-05 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.516061+0000", 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8672999999999999e-05 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.301 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:91918 7.e 7:7457bb15:::benchmark_data_smithi055_81491_object91917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:54.469385+0000", 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.733156669, 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.046871696999999997, 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91918 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469385+0000", 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469385+0000", 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469385+0000", 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8599999999999997e-07 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469481+0000", 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5322000000000007e-05 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469490+0000", 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8400000000000001e-06 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.469497+0000", 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8590000000000005e-06 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.495029+0000", 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.025531814 2024-03-26T21:31:56.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.495160+0000", 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013034400000000001 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.495294+0000", 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013406599999999999 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504068+0000", 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087741710000000007 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504156+0000", 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7675000000000003e-05 2024-03-26T21:31:56.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.516086+0000", 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011930186000000001 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.516223+0000", 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013757 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.516256+0000", 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3064000000000001e-05 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:91921 7.e 7:71ef6414:::benchmark_data_smithi055_81491_object91920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:54.470191+0000", 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.7323501509999999, 2024-03-26T21:31:56.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.047103006000000003, 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91921 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.310 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470191+0000", 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470191+0000", 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470192+0000", 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1519999999999999e-06 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470224+0000", 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2051000000000002e-05 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470235+0000", 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0757000000000001e-05 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470244+0000", 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2180000000000005e-06 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.499934+0000", 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.029689329 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.500106+0000", 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017188699999999999 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.500272+0000", 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016621100000000001 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504196+0000", 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0039241010000000002 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.315 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.511657+0000", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0074612350000000001 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517014+0000", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0053566229999999996 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517252+0000", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023823 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517294+0000", 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2212000000000002e-05 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:91922 7.e 7:737c9866:::benchmark_data_smithi055_81491_object91921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:31:54.470258+0000", 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.732283716, 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.047301139999999998, 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:31:56.317 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91922 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.318 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470258+0000", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470258+0000", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470258+0000", 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8299999999999997e-07 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470278+0000", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9502000000000001e-05 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470284+0000", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5400000000000001e-06 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.470291+0000", 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3799999999999999e-06 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504232+0000", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.033941856999999999 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504357+0000", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000124783 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,5", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.504506+0000", 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014919999999999999 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.511604+0000", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0070978150000000004 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.511702+0000", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8275000000000004e-05 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517304+0000", 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056019529999999998 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.322 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517516+0000", 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021198699999999999 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:31:54.517559+0000", 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2364999999999998e-05 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:31:56.323 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:31:56.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:56.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:56.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:56.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:56.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:56.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:56.710 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 63 16 94201 94185 93.4175 61.375 0.0124017 0.0106993 2024-03-26T21:31:56.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:56.794 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:56.896 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:56.948 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:56.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:56.957 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:56 smithi055 ceph-mon[40587]: pgmap v116: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 903 op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:56.957 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:56 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 27808/266562 objects degraded (10.432%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:56.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:56 smithi055 ceph-mon[29406]: pgmap v116: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 903 op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:56.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:56 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 27808/266562 objects degraded (10.432%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:56 smithi067 ceph-mon[33346]: pgmap v116: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 903 op/s; 27808/266562 objects degraded (10.432%) 2024-03-26T21:31:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:56 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 27808/266562 objects degraded (10.432%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:31:56.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:31:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:31:57.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:57.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:57.193 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:57.232 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:57.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:57.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:57.365 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:57.382 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:31:57.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:31:57.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:57.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:31:57.637 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:31:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.659 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:57.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:31:57.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 64 16 95159 95143 92.8926 59.875 0.00948225 0.0107595 2024-03-26T21:31:57.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.808 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:57.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:57.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:57.949 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:57.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2037145483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:57 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:31:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:57 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:31:57.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:57.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2037145483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:31:57.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:31:57.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:31:57.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2037145483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:31:57.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:31:57.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:57 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:31:58.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:58.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:58.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:58.208 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:58.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:31:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:31:58.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:58.478 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:58.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:58.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:58.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 65 16 96294 96278 92.5548 70.9375 0.0101495 0.0107997 2024-03-26T21:31:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[29406]: pgmap v117: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[29406]: osdmap e116: 8 total, 7 up, 8 in 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[40587]: pgmap v117: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:58.809 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:58.810 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-mon[40587]: osdmap e116: 8 total, 7 up, 8 in 2024-03-26T21:31:58.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:31:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:58.884 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:58.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:31:58.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:31:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:58 smithi067 ceph-mon[33346]: pgmap v117: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:31:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:58 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:31:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:31:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:31:58 smithi067 ceph-mon[33346]: osdmap e116: 8 total, 7 up, 8 in 2024-03-26T21:31:59.024 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:31:59.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:31:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:31:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:31:59.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:59.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:59.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:31:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.301 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:31:59.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:31:59.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:59.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:31:59.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:59.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 66 16 97369 97353 92.1703 67.1875 0.00610618 0.010844 2024-03-26T21:31:59.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:31:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:31:59.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:31:59.946 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:31:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:31:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:00.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:00.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:00.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:00.205 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:00.307 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:00.345 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:00.422 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:00.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.508 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:00.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:00.609 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.658 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:00.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:00.665 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[29406]: pgmap v119: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:32:00.665 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:00.665 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:00.665 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[29406]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T21:32:00.665 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:00.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 67 16 98469 98453 91.8207 68.75 0.014106 0.0108861 2024-03-26T21:32:00.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:00.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:00.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:32:00.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[40587]: pgmap v119: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:32:00.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:00.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:00.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[40587]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T21:32:00.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:00.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:00.944 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:00.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:00 smithi067 ceph-mon[33346]: pgmap v119: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 29456/282270 objects degraded (10.435%) 2024-03-26T21:32:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:00 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:00 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:00 smithi067 ceph-mon[33346]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T21:32:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:01.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:01.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:01.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:01.257 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:01.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:01.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:01.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:01.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:01.655 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:01.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 68 16 99448 99432 91.3702 61.1875 0.0129718 0.0109392 2024-03-26T21:32:01.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:01.757 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3440634040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:01.757 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:01 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 29456/282270 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:01.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3440634040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:01.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:01 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 29456/282270 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:01.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:01.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:01.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:01.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3440634040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:01 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 29456/282270 objects degraded (10.435%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:02.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:02.065 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:02.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:02.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:02.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:02.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:02.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:02.416 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:02.416 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:02.416 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:02.416 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:02.416 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:32:02.417 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:02.417 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:02.417 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:02.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:02.489 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:02.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:02.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 69 16 100547 100531 91.0414 68.6875 0.0166802 0.0109793 2024-03-26T21:32:02.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:02.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:02.894 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:02.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:02 smithi067 ceph-mon[33346]: pgmap v121: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s; 29456/287085 objects degraded (10.260%) 2024-03-26T21:32:02.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:03.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:02 smithi055 ceph-mon[40587]: pgmap v121: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s; 29456/287085 objects degraded (10.260%) 2024-03-26T21:32:03.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:02 smithi055 ceph-mon[29406]: pgmap v121: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s; 29456/287085 objects degraded (10.260%) 2024-03-26T21:32:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:32:03.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:32:03.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:03.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:03.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.301 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:03.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:03.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:03.426 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.433 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:03.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:03.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.584 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:03.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:03.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:03.711 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 70 16 101590 101574 90.672 65.1875 0.0215741 0.0110238 2024-03-26T21:32:03.726 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:03.828 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:03.873 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:32:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:03.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:03.877 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:32:03.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.022 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:04.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.163 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:04.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.313 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:04.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:04.455 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:04.495 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:32:04.496 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:32:04.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:04.593 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:04.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:04.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 71 16 102532 102516 90.2242 58.875 0.0124913 0.0110789 2024-03-26T21:32:04.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:04.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:04.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:04 smithi067 ceph-mon[33346]: pgmap v122: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:04 smithi067 ceph-mon[33346]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:04.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:04.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:05.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:04 smithi055 ceph-mon[40587]: pgmap v122: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:05.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:04 smithi055 ceph-mon[40587]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:04 smithi055 ceph-mon[29406]: pgmap v122: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:04 smithi055 ceph-mon[29406]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:05.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:05.111 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:32:05.137 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:05.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:05.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.416 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:05.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:05.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:05.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:05.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.673 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:05.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:05.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 72 16 103556 103540 89.8599 64 0.0245643 0.0111232 2024-03-26T21:32:05.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:05.815 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:05.866 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:32:05.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:05.972 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:05.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:06.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:06.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:06.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:06.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:06.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:06.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:06.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:06.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:06.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 73 16 104604 104588 89.5262 65.5 0.0085828 0.0111652 2024-03-26T21:32:06.778 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-26T21:32:03.329838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6497442216,"num_objects":99332,"num_object_clones":0,"num_object_copies":297996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99332,"num_whiteouts":0,"num_read":1126,"num_read_kb":972,"num_write":99749,"num_write_kb":6347037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88878,"ondisk_log_size":88878,"up":381,"acting":381,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":439,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17527748,"kb_used_data":16954864,"kb_used_omap":12,"kb_used_meta":572787,"kb_avail":732433468,"statfs":{"total":767960285184,"available":750011871232,"internally_reserved":0,"allocated":17361780736,"data_stored":17358147972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":586534448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,5,7,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":686686208,"num_objects":10478,"num_object_clones":0,"num_object_copies":31434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10478,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10478,"num_write_kb":670592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006757"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.688012+0000","last_change":"2024-03-26T21:30:31.738275+0000","last_active":"2024-03-26T21:31:59.688012+0000","last_peered":"2024-03-26T21:31:59.688012+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738275+0000","last_became_peered":"2024-03-26T21:30:31.738275+0000","last_unstale":"2024-03-26T21:31:59.688012+0000","last_undegraded":"2024-03-26T21:31:59.688012+0000","last_fullsized":"2024-03-26T21:30:31.732719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.686394+0000","last_change":"2024-03-26T21:30:12.067262+0000","last_active":"2024-03-26T21:31:59.686394+0000","last_peered":"2024-03-26T21:31:59.686394+0000","last_clean":"2024-03-26T21:31:59.686394+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:31:59.686394+0000","last_undegraded":"2024-03-26T21:31:59.686394+0000","last_fullsized":"2024-03-26T21:31:59.686394+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.067176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.067176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.675406+0000","last_change":"2024-03-26T21:30:31.663369+0000","last_active":"2024-03-26T21:31:59.675406+0000","last_peered":"2024-03-26T21:31:59.675406+0000","last_clean":"2024-03-26T21:31:59.675406+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:31:59.675406+0000","last_undegraded":"2024-03-26T21:31:59.675406+0000","last_fullsized":"2024-03-26T21:31:59.675406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:30:31.663239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":90,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.675246+0000","last_change":"2024-03-26T21:30:31.740069+0000","last_active":"2024-03-26T21:31:59.675246+0000","last_peered":"2024-03-26T21:31:59.675246+0000","last_clean":"2024-03-26T21:28:42.030750+0000","last_became_active":"2024-03-26T21:30:31.740069+0000","last_became_peered":"2024-03-26T21:30:31.740069+0000","last_unstale":"2024-03-26T21:31:59.675246+0000","last_undegraded":"2024-03-26T21:30:31.731923+0000","last_fullsized":"2024-03-26T21:30:31.731769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681213+0000","last_change":"2024-03-26T21:31:16.924155+0000","last_active":"2024-03-26T21:31:59.681213+0000","last_peered":"2024-03-26T21:31:59.681213+0000","last_clean":"2024-03-26T21:31:59.681213+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:31:59.681213+0000","last_undegraded":"2024-03-26T21:31:59.681213+0000","last_fullsized":"2024-03-26T21:31:59.681213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:16.923867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:16.923867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:27.606830+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.674901+0000","last_change":"2024-03-26T21:30:31.739659+0000","last_active":"2024-03-26T21:31:59.674901+0000","last_peered":"2024-03-26T21:31:59.674901+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.739659+0000","last_became_peered":"2024-03-26T21:30:31.739659+0000","last_unstale":"2024-03-26T21:31:59.674901+0000","last_undegraded":"2024-03-26T21:31:59.674901+0000","last_fullsized":"2024-03-26T21:30:31.731768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.682058+0000","last_change":"2024-03-26T21:31:17.917878+0000","last_active":"2024-03-26T21:31:59.682058+0000","last_peered":"2024-03-26T21:31:59.682058+0000","last_clean":"2024-03-26T21:31:59.682058+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:31:59.682058+0000","last_undegraded":"2024-03-26T21:31:59.682058+0000","last_fullsized":"2024-03-26T21:31:59.682058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:17.917762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:17.917762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.999530+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.680657+0000","last_change":"2024-03-26T21:30:05.728263+0000","last_active":"2024-03-26T21:31:59.680657+0000","last_peered":"2024-03-26T21:31:59.680657+0000","last_clean":"2024-03-26T21:31:59.680657+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:31:59.680657+0000","last_undegraded":"2024-03-26T21:31:59.680657+0000","last_fullsized":"2024-03-26T21:31:59.680657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:05.728097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.728097+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674073+0000","last_change":"2024-03-26T21:31:42.540278+0000","last_active":"2024-03-26T21:31:59.674073+0000","last_peered":"2024-03-26T21:31:59.674073+0000","last_clean":"2024-03-26T21:31:59.674073+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:31:59.674073+0000","last_undegraded":"2024-03-26T21:31:59.674073+0000","last_fullsized":"2024-03-26T21:31:59.674073+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:42.540161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:42.540161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:01.295047+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.690081+0000","last_change":"2024-03-26T21:30:23.539609+0000","last_active":"2024-03-26T21:31:59.690081+0000","last_peered":"2024-03-26T21:31:59.690081+0000","last_clean":"2024-03-26T21:31:59.690081+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:31:59.690081+0000","last_undegraded":"2024-03-26T21:31:59.690081+0000","last_fullsized":"2024-03-26T21:31:59.690081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:30:23.539529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.689021+0000","last_change":"2024-03-26T21:30:31.738978+0000","last_active":"2024-03-26T21:31:59.689021+0000","last_peered":"2024-03-26T21:31:59.689021+0000","last_clean":"2024-03-26T21:30:22.526419+0000","last_became_active":"2024-03-26T21:30:31.738978+0000","last_became_peered":"2024-03-26T21:30:31.738978+0000","last_unstale":"2024-03-26T21:31:59.689021+0000","last_undegraded":"2024-03-26T21:31:59.689021+0000","last_fullsized":"2024-03-26T21:30:31.733643+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:22.526329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:22.526329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":90,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.680707+0000","last_change":"2024-03-26T21:30:31.742645+0000","last_active":"2024-03-26T21:31:59.680707+0000","last_peered":"2024-03-26T21:31:59.680707+0000","last_clean":"2024-03-26T21:28:42.145195+0000","last_became_active":"2024-03-26T21:30:31.742645+0000","last_became_peered":"2024-03-26T21:30:31.742645+0000","last_unstale":"2024-03-26T21:31:59.680707+0000","last_undegraded":"2024-03-26T21:30:31.730765+0000","last_fullsized":"2024-03-26T21:30:31.730641+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687800+0000","last_change":"2024-03-26T21:30:11.468492+0000","last_active":"2024-03-26T21:31:59.687800+0000","last_peered":"2024-03-26T21:31:59.687800+0000","last_clean":"2024-03-26T21:31:59.687800+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:31:59.687800+0000","last_undegraded":"2024-03-26T21:31:59.687800+0000","last_fullsized":"2024-03-26T21:31:59.687800+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.468385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.669276+0000","last_change":"2024-03-26T21:30:31.744378+0000","last_active":"2024-03-26T21:31:59.669276+0000","last_peered":"2024-03-26T21:31:59.669276+0000","last_clean":"2024-03-26T21:30:19.144400+0000","last_became_active":"2024-03-26T21:30:31.744378+0000","last_became_peered":"2024-03-26T21:30:31.744378+0000","last_unstale":"2024-03-26T21:31:59.669276+0000","last_undegraded":"2024-03-26T21:31:59.669276+0000","last_fullsized":"2024-03-26T21:30:31.735352+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:07.539633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.539633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.688436+0000","last_change":"2024-03-26T21:30:31.738170+0000","last_active":"2024-03-26T21:31:59.688436+0000","last_peered":"2024-03-26T21:31:59.688436+0000","last_clean":"2024-03-26T21:30:28.546608+0000","last_became_active":"2024-03-26T21:30:31.738170+0000","last_became_peered":"2024-03-26T21:30:31.738170+0000","last_unstale":"2024-03-26T21:31:59.688436+0000","last_undegraded":"2024-03-26T21:31:59.688436+0000","last_fullsized":"2024-03-26T21:30:31.731908+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:28.546501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:28.546501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":84,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.680391+0000","last_change":"2024-03-26T21:30:31.745243+0000","last_active":"2024-03-26T21:31:59.680391+0000","last_peered":"2024-03-26T21:31:59.680391+0000","last_clean":"2024-03-26T21:28:42.123479+0000","last_became_active":"2024-03-26T21:30:31.745243+0000","last_became_peered":"2024-03-26T21:30:31.745243+0000","last_unstale":"2024-03-26T21:31:59.680391+0000","last_undegraded":"2024-03-26T21:30:31.731435+0000","last_fullsized":"2024-03-26T21:30:31.731307+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.676514+0000","last_change":"2024-03-26T21:30:31.740216+0000","last_active":"2024-03-26T21:31:59.676514+0000","last_peered":"2024-03-26T21:31:59.676514+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.740216+0000","last_became_peered":"2024-03-26T21:30:31.740216+0000","last_unstale":"2024-03-26T21:31:59.676514+0000","last_undegraded":"2024-03-26T21:31:59.676514+0000","last_fullsized":"2024-03-26T21:30:31.732999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.687708+0000","last_change":"2024-03-26T21:30:31.748769+0000","last_active":"2024-03-26T21:31:59.687708+0000","last_peered":"2024-03-26T21:31:59.687708+0000","last_clean":"2024-03-26T21:30:19.057549+0000","last_became_active":"2024-03-26T21:30:31.748769+0000","last_became_peered":"2024-03-26T21:30:31.748769+0000","last_unstale":"2024-03-26T21:31:59.687708+0000","last_undegraded":"2024-03-26T21:31:59.687708+0000","last_fullsized":"2024-03-26T21:30:31.734303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.051556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.051556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.674637+0000","last_change":"2024-03-26T21:30:31.744225+0000","last_active":"2024-03-26T21:31:59.674637+0000","last_peered":"2024-03-26T21:31:59.674637+0000","last_clean":"2024-03-26T21:30:19.056406+0000","last_became_active":"2024-03-26T21:30:31.744225+0000","last_became_peered":"2024-03-26T21:30:31.744225+0000","last_unstale":"2024-03-26T21:31:59.674637+0000","last_undegraded":"2024-03-26T21:31:59.674637+0000","last_fullsized":"2024-03-26T21:30:31.737824+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.708767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.708767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":149,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.678017+0000","last_change":"2024-03-26T21:31:18.114712+0000","last_active":"2024-03-26T21:31:59.678017+0000","last_peered":"2024-03-26T21:31:59.678017+0000","last_clean":"2024-03-26T21:31:59.678017+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:31:59.678017+0000","last_undegraded":"2024-03-26T21:31:59.678017+0000","last_fullsized":"2024-03-26T21:31:59.678017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:31:18.114493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:18.114493+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:28.564612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679647+0000","last_change":"2024-03-26T21:30:32.797018+0000","last_active":"2024-03-26T21:31:59.679647+0000","last_peered":"2024-03-26T21:31:59.679647+0000","last_clean":"2024-03-26T21:31:59.679647+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:31:59.679647+0000","last_undegraded":"2024-03-26T21:31:59.679647+0000","last_fullsized":"2024-03-26T21:31:59.679647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:32.796920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:32.796920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.681212+0000","last_change":"2024-03-26T21:30:31.741526+0000","last_active":"2024-03-26T21:31:59.681212+0000","last_peered":"2024-03-26T21:31:59.681212+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.741526+0000","last_became_peered":"2024-03-26T21:30:31.741526+0000","last_unstale":"2024-03-26T21:31:59.681212+0000","last_undegraded":"2024-03-26T21:31:59.681212+0000","last_fullsized":"2024-03-26T21:30:31.731261+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":56,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.686696+0000","last_change":"2024-03-26T21:30:31.740626+0000","last_active":"2024-03-26T21:31:59.686696+0000","last_peered":"2024-03-26T21:31:59.686696+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740626+0000","last_became_peered":"2024-03-26T21:30:31.740626+0000","last_unstale":"2024-03-26T21:31:59.686696+0000","last_undegraded":"2024-03-26T21:31:59.686696+0000","last_fullsized":"2024-03-26T21:30:31.732923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":134,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.680611+0000","last_change":"2024-03-26T21:30:31.745104+0000","last_active":"2024-03-26T21:31:59.680611+0000","last_peered":"2024-03-26T21:31:59.680611+0000","last_clean":"2024-03-26T21:30:19.791949+0000","last_became_active":"2024-03-26T21:30:31.745104+0000","last_became_peered":"2024-03-26T21:30:31.745104+0000","last_unstale":"2024-03-26T21:31:59.680611+0000","last_undegraded":"2024-03-26T21:30:31.729821+0000","last_fullsized":"2024-03-26T21:30:31.729686+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.670583+0000","last_change":"2024-03-26T21:31:14.781495+0000","last_active":"2024-03-26T21:31:59.670583+0000","last_peered":"2024-03-26T21:31:59.670583+0000","last_clean":"2024-03-26T21:31:59.670583+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:31:59.670583+0000","last_undegraded":"2024-03-26T21:31:59.670583+0000","last_fullsized":"2024-03-26T21:31:59.670583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:14.781351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.781351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.404865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":169,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.682006+0000","last_change":"2024-03-26T21:30:31.738596+0000","last_active":"2024-03-26T21:31:59.682006+0000","last_peered":"2024-03-26T21:31:59.682006+0000","last_clean":"2024-03-26T21:30:29.906831+0000","last_became_active":"2024-03-26T21:30:31.738596+0000","last_became_peered":"2024-03-26T21:30:31.738596+0000","last_unstale":"2024-03-26T21:31:59.682006+0000","last_undegraded":"2024-03-26T21:30:31.733688+0000","last_fullsized":"2024-03-26T21:30:31.733580+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:10.722203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.722203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":125,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.686488+0000","last_change":"2024-03-26T21:30:31.740769+0000","last_active":"2024-03-26T21:31:59.686488+0000","last_peered":"2024-03-26T21:31:59.686488+0000","last_clean":"2024-03-26T21:30:19.055952+0000","last_became_active":"2024-03-26T21:30:31.740769+0000","last_became_peered":"2024-03-26T21:30:31.740769+0000","last_unstale":"2024-03-26T21:31:59.686488+0000","last_undegraded":"2024-03-26T21:30:31.730676+0000","last_fullsized":"2024-03-26T21:30:31.730563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.058714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687056+0000","last_change":"2024-03-26T21:30:37.614796+0000","last_active":"2024-03-26T21:31:59.687056+0000","last_peered":"2024-03-26T21:31:59.687056+0000","last_clean":"2024-03-26T21:31:59.687056+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:31:59.687056+0000","last_undegraded":"2024-03-26T21:31:59.687056+0000","last_fullsized":"2024-03-26T21:31:59.687056+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:37.614664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.614664+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.687174+0000","last_change":"2024-03-26T21:30:31.739460+0000","last_active":"2024-03-26T21:31:59.687174+0000","last_peered":"2024-03-26T21:31:59.687174+0000","last_clean":"2024-03-26T21:30:19.057225+0000","last_became_active":"2024-03-26T21:30:31.739460+0000","last_became_peered":"2024-03-26T21:30:31.739460+0000","last_unstale":"2024-03-26T21:31:59.687174+0000","last_undegraded":"2024-03-26T21:31:59.687174+0000","last_fullsized":"2024-03-26T21:30:31.733695+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687462+0000","last_change":"2024-03-26T21:31:28.707417+0000","last_active":"2024-03-26T21:31:59.687462+0000","last_peered":"2024-03-26T21:31:59.687462+0000","last_clean":"2024-03-26T21:31:59.687462+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:31:59.687462+0000","last_undegraded":"2024-03-26T21:31:59.687462+0000","last_fullsized":"2024-03-26T21:31:59.687462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.707018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.375332+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.678829+0000","last_change":"2024-03-26T21:31:39.883235+0000","last_active":"2024-03-26T21:31:59.678829+0000","last_peered":"2024-03-26T21:31:59.678829+0000","last_clean":"2024-03-26T21:31:59.678829+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:31:59.678829+0000","last_undegraded":"2024-03-26T21:31:59.678829+0000","last_fullsized":"2024-03-26T21:31:59.678829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.883105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.883105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:03.189316+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681921+0000","last_change":"2024-03-26T21:31:23.609409+0000","last_active":"2024-03-26T21:31:59.681921+0000","last_peered":"2024-03-26T21:31:59.681921+0000","last_clean":"2024-03-26T21:31:59.681921+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:31:59.681921+0000","last_undegraded":"2024-03-26T21:31:59.681921+0000","last_fullsized":"2024-03-26T21:31:59.681921+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:31:23.609259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:23.609259+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:34.316708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.669087+0000","last_change":"2024-03-26T21:30:31.738804+0000","last_active":"2024-03-26T21:31:59.669087+0000","last_peered":"2024-03-26T21:31:59.669087+0000","last_clean":"2024-03-26T21:30:19.144401+0000","last_became_active":"2024-03-26T21:30:31.738804+0000","last_became_peered":"2024-03-26T21:30:31.738804+0000","last_unstale":"2024-03-26T21:31:59.669087+0000","last_undegraded":"2024-03-26T21:31:59.669087+0000","last_fullsized":"2024-03-26T21:30:31.735055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.535399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.535399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.676121+0000","last_change":"2024-03-26T21:30:31.739828+0000","last_active":"2024-03-26T21:31:59.676121+0000","last_peered":"2024-03-26T21:31:59.676121+0000","last_clean":"2024-03-26T21:30:19.905962+0000","last_became_active":"2024-03-26T21:30:31.739828+0000","last_became_peered":"2024-03-26T21:30:31.739828+0000","last_unstale":"2024-03-26T21:31:59.676121+0000","last_undegraded":"2024-03-26T21:31:59.676121+0000","last_fullsized":"2024-03-26T21:30:31.731999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.749046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.749046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.675575+0000","last_change":"2024-03-26T21:31:26.716901+0000","last_active":"2024-03-26T21:31:59.675575+0000","last_peered":"2024-03-26T21:31:59.675575+0000","last_clean":"2024-03-26T21:31:59.675575+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:31:59.675575+0000","last_undegraded":"2024-03-26T21:31:59.675575+0000","last_fullsized":"2024-03-26T21:31:59.675575+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:26.716779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:26.716779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.514181+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":130,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.678932+0000","last_change":"2024-03-26T21:31:25.930138+0000","last_active":"2024-03-26T21:31:59.678932+0000","last_peered":"2024-03-26T21:31:59.678932+0000","last_clean":"2024-03-26T21:31:59.678932+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:31:59.678932+0000","last_undegraded":"2024-03-26T21:31:59.678932+0000","last_fullsized":"2024-03-26T21:31:59.678932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:31:25.930008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.930008+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.183279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681607+0000","last_change":"2024-03-26T21:30:29.844822+0000","last_active":"2024-03-26T21:31:59.681607+0000","last_peered":"2024-03-26T21:31:59.681607+0000","last_clean":"2024-03-26T21:31:59.681607+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:31:59.681607+0000","last_undegraded":"2024-03-26T21:31:59.681607+0000","last_fullsized":"2024-03-26T21:31:59.681607+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:30:29.844710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":107,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.669561+0000","last_change":"2024-03-26T21:30:31.742998+0000","last_active":"2024-03-26T21:31:59.669561+0000","last_peered":"2024-03-26T21:31:59.669561+0000","last_clean":"2024-03-26T21:30:19.144660+0000","last_became_active":"2024-03-26T21:30:31.742998+0000","last_became_peered":"2024-03-26T21:30:31.742998+0000","last_unstale":"2024-03-26T21:31:59.669561+0000","last_undegraded":"2024-03-26T21:31:59.669561+0000","last_fullsized":"2024-03-26T21:30:31.735078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688476+0000","last_change":"2024-03-26T21:31:37.579162+0000","last_active":"2024-03-26T21:31:59.688476+0000","last_peered":"2024-03-26T21:31:59.688476+0000","last_clean":"2024-03-26T21:31:59.688476+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:31:59.688476+0000","last_undegraded":"2024-03-26T21:31:59.688476+0000","last_fullsized":"2024-03-26T21:31:59.688476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.579051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.579051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:43.787885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":135,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.680556+0000","last_change":"2024-03-26T21:30:18.768249+0000","last_active":"2024-03-26T21:31:59.680556+0000","last_peered":"2024-03-26T21:31:59.680556+0000","last_clean":"2024-03-26T21:31:59.680556+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:31:59.680556+0000","last_undegraded":"2024-03-26T21:31:59.680556+0000","last_fullsized":"2024-03-26T21:31:59.680556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:18.768133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:18.768133+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.673835+0000","last_change":"2024-03-26T21:31:22.719019+0000","last_active":"2024-03-26T21:31:59.673835+0000","last_peered":"2024-03-26T21:31:59.673835+0000","last_clean":"2024-03-26T21:31:59.673835+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:31:59.673835+0000","last_undegraded":"2024-03-26T21:31:59.673835+0000","last_fullsized":"2024-03-26T21:31:59.673835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:22.718906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:22.718906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:23.052146+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681019+0000","last_change":"2024-03-26T21:31:32.871165+0000","last_active":"2024-03-26T21:31:59.681019+0000","last_peered":"2024-03-26T21:31:59.681019+0000","last_clean":"2024-03-26T21:31:59.681019+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:31:59.681019+0000","last_undegraded":"2024-03-26T21:31:59.681019+0000","last_fullsized":"2024-03-26T21:31:59.681019+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:32.871039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.871039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:53.774307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.671443+0000","last_change":"2024-03-26T21:30:31.736143+0000","last_active":"2024-03-26T21:31:59.671443+0000","last_peered":"2024-03-26T21:31:59.671443+0000","last_clean":"2024-03-26T21:30:19.144685+0000","last_became_active":"2024-03-26T21:30:31.736143+0000","last_became_peered":"2024-03-26T21:30:31.736143+0000","last_unstale":"2024-03-26T21:31:59.671443+0000","last_undegraded":"2024-03-26T21:31:59.671443+0000","last_fullsized":"2024-03-26T21:30:31.733617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687212+0000","last_change":"2024-03-26T21:31:36.082385+0000","last_active":"2024-03-26T21:31:59.687212+0000","last_peered":"2024-03-26T21:31:59.687212+0000","last_clean":"2024-03-26T21:31:59.687212+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:31:59.687212+0000","last_undegraded":"2024-03-26T21:31:59.687212+0000","last_fullsized":"2024-03-26T21:31:59.687212+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:31:36.082236+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:31:36.082236+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.095627+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.680000+0000","last_change":"2024-03-26T21:31:38.898103+0000","last_active":"2024-03-26T21:31:59.680000+0000","last_peered":"2024-03-26T21:31:59.680000+0000","last_clean":"2024-03-26T21:31:59.680000+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:31:59.680000+0000","last_undegraded":"2024-03-26T21:31:59.680000+0000","last_fullsized":"2024-03-26T21:31:59.680000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:38.897810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.897810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:50.848439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687042+0000","last_change":"2024-03-26T21:31:33.046592+0000","last_active":"2024-03-26T21:31:59.687042+0000","last_peered":"2024-03-26T21:31:59.687042+0000","last_clean":"2024-03-26T21:31:59.687042+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:31:59.687042+0000","last_undegraded":"2024-03-26T21:31:59.687042+0000","last_fullsized":"2024-03-26T21:31:59.687042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:33.046470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:33.046470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.166286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":56,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.670041+0000","last_change":"2024-03-26T21:30:31.741210+0000","last_active":"2024-03-26T21:31:59.670041+0000","last_peered":"2024-03-26T21:31:59.670041+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.741210+0000","last_became_peered":"2024-03-26T21:30:31.741210+0000","last_unstale":"2024-03-26T21:31:59.670041+0000","last_undegraded":"2024-03-26T21:31:59.670041+0000","last_fullsized":"2024-03-26T21:30:31.734566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:26.155650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:26.155650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":93,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.675702+0000","last_change":"2024-03-26T21:30:31.741165+0000","last_active":"2024-03-26T21:31:59.675702+0000","last_peered":"2024-03-26T21:31:59.675702+0000","last_clean":"2024-03-26T21:28:42.207254+0000","last_became_active":"2024-03-26T21:30:31.741165+0000","last_became_peered":"2024-03-26T21:30:31.741165+0000","last_unstale":"2024-03-26T21:31:59.675702+0000","last_undegraded":"2024-03-26T21:30:31.733400+0000","last_fullsized":"2024-03-26T21:30:31.733163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.686576+0000","last_change":"2024-03-26T21:31:40.605350+0000","last_active":"2024-03-26T21:31:59.686576+0000","last_peered":"2024-03-26T21:31:59.686576+0000","last_clean":"2024-03-26T21:31:59.686576+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:31:59.686576+0000","last_undegraded":"2024-03-26T21:31:59.686576+0000","last_fullsized":"2024-03-26T21:31:59.686576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:40.605038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:40.605038+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.927678+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":171,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688909+0000","last_change":"2024-03-26T21:31:19.736062+0000","last_active":"2024-03-26T21:31:59.688909+0000","last_peered":"2024-03-26T21:31:59.688909+0000","last_clean":"2024-03-26T21:31:59.688909+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:31:59.688909+0000","last_undegraded":"2024-03-26T21:31:59.688909+0000","last_fullsized":"2024-03-26T21:31:59.688909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:31:19.735944+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:31:19.735944+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.497485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.669688+0000","last_change":"2024-03-26T21:30:31.738173+0000","last_active":"2024-03-26T21:31:59.669688+0000","last_peered":"2024-03-26T21:31:59.669688+0000","last_clean":"2024-03-26T21:30:19.144537+0000","last_became_active":"2024-03-26T21:30:31.738173+0000","last_became_peered":"2024-03-26T21:30:31.738173+0000","last_unstale":"2024-03-26T21:31:59.669688+0000","last_undegraded":"2024-03-26T21:31:59.669688+0000","last_fullsized":"2024-03-26T21:30:31.733359+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681571+0000","last_change":"2024-03-26T21:31:28.999375+0000","last_active":"2024-03-26T21:31:59.681571+0000","last_peered":"2024-03-26T21:31:59.681571+0000","last_clean":"2024-03-26T21:31:59.681571+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:31:59.681571+0000","last_undegraded":"2024-03-26T21:31:59.681571+0000","last_fullsized":"2024-03-26T21:31:59.681571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:31:28.998501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.998501+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.965671+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.678791+0000","last_change":"2024-03-26T21:30:31.744131+0000","last_active":"2024-03-26T21:31:59.678791+0000","last_peered":"2024-03-26T21:31:59.678791+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.744131+0000","last_became_peered":"2024-03-26T21:30:31.744131+0000","last_unstale":"2024-03-26T21:31:59.678791+0000","last_undegraded":"2024-03-26T21:31:59.678791+0000","last_fullsized":"2024-03-26T21:30:31.735473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":165,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681800+0000","last_change":"2024-03-26T21:30:37.766284+0000","last_active":"2024-03-26T21:31:59.681800+0000","last_peered":"2024-03-26T21:31:59.681800+0000","last_clean":"2024-03-26T21:31:59.681800+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:31:59.681800+0000","last_undegraded":"2024-03-26T21:31:59.681800+0000","last_fullsized":"2024-03-26T21:31:59.681800+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:37.766196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.766196+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:07.457096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674006+0000","last_change":"2024-03-26T21:30:33.644993+0000","last_active":"2024-03-26T21:31:59.674006+0000","last_peered":"2024-03-26T21:31:59.674006+0000","last_clean":"2024-03-26T21:31:59.674006+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:31:59.674006+0000","last_undegraded":"2024-03-26T21:31:59.674006+0000","last_fullsized":"2024-03-26T21:31:59.674006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:33.644876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:33.644876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.625879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":140,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.689134+0000","last_change":"2024-03-26T21:31:14.634077+0000","last_active":"2024-03-26T21:31:59.689134+0000","last_peered":"2024-03-26T21:31:59.689134+0000","last_clean":"2024-03-26T21:31:59.689134+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:31:59.689134+0000","last_undegraded":"2024-03-26T21:31:59.689134+0000","last_fullsized":"2024-03-26T21:31:59.689134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:31:14.633936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.633936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:30.906840+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.678292+0000","last_change":"2024-03-26T21:30:31.739981+0000","last_active":"2024-03-26T21:31:59.678292+0000","last_peered":"2024-03-26T21:31:59.678292+0000","last_clean":"2024-03-26T21:30:19.265055+0000","last_became_active":"2024-03-26T21:30:31.739981+0000","last_became_peered":"2024-03-26T21:30:31.739981+0000","last_unstale":"2024-03-26T21:31:59.678292+0000","last_undegraded":"2024-03-26T21:31:59.678292+0000","last_fullsized":"2024-03-26T21:30:31.736585+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.958658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.958658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.680153+0000","last_change":"2024-03-26T21:30:24.865430+0000","last_active":"2024-03-26T21:31:59.680153+0000","last_peered":"2024-03-26T21:31:59.680153+0000","last_clean":"2024-03-26T21:31:59.680153+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:31:59.680153+0000","last_undegraded":"2024-03-26T21:31:59.680153+0000","last_fullsized":"2024-03-26T21:31:59.680153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:24.865340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:24.865340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.669132+0000","last_change":"2024-03-26T21:30:31.742874+0000","last_active":"2024-03-26T21:31:59.669132+0000","last_peered":"2024-03-26T21:31:59.669132+0000","last_clean":"2024-03-26T21:30:30.545271+0000","last_became_active":"2024-03-26T21:30:31.742874+0000","last_became_peered":"2024-03-26T21:30:31.742874+0000","last_unstale":"2024-03-26T21:31:59.669132+0000","last_undegraded":"2024-03-26T21:31:59.669132+0000","last_fullsized":"2024-03-26T21:30:31.735756+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:30.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:30.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687297+0000","last_change":"2024-03-26T21:31:38.685593+0000","last_active":"2024-03-26T21:31:59.687297+0000","last_peered":"2024-03-26T21:31:59.687297+0000","last_clean":"2024-03-26T21:31:59.687297+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:31:59.687297+0000","last_undegraded":"2024-03-26T21:31:59.687297+0000","last_fullsized":"2024-03-26T21:31:59.687297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.685453+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.982628+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.671366+0000","last_change":"2024-03-26T21:30:31.742256+0000","last_active":"2024-03-26T21:31:59.671366+0000","last_peered":"2024-03-26T21:31:59.671366+0000","last_clean":"2024-03-26T21:30:27.542711+0000","last_became_active":"2024-03-26T21:30:31.742256+0000","last_became_peered":"2024-03-26T21:30:31.742256+0000","last_unstale":"2024-03-26T21:31:59.671366+0000","last_undegraded":"2024-03-26T21:31:59.671366+0000","last_fullsized":"2024-03-26T21:30:31.733298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:30:27.542602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688709+0000","last_change":"2024-03-26T21:31:27.662639+0000","last_active":"2024-03-26T21:31:59.688709+0000","last_peered":"2024-03-26T21:31:59.688709+0000","last_clean":"2024-03-26T21:31:59.688709+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:31:59.688709+0000","last_undegraded":"2024-03-26T21:31:59.688709+0000","last_fullsized":"2024-03-26T21:31:59.688709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:27.662555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:27.662555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:29.340810+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674796+0000","last_change":"2024-03-26T21:31:31.675402+0000","last_active":"2024-03-26T21:31:59.674796+0000","last_peered":"2024-03-26T21:31:59.674796+0000","last_clean":"2024-03-26T21:31:59.674796+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:31:59.674796+0000","last_undegraded":"2024-03-26T21:31:59.674796+0000","last_fullsized":"2024-03-26T21:31:59.674796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:31.675298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:31.675298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:55.883626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679972+0000","last_change":"2024-03-26T21:30:09.668099+0000","last_active":"2024-03-26T21:31:59.679972+0000","last_peered":"2024-03-26T21:31:59.679972+0000","last_clean":"2024-03-26T21:31:59.679972+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:31:59.679972+0000","last_undegraded":"2024-03-26T21:31:59.679972+0000","last_fullsized":"2024-03-26T21:31:59.679972+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:30:09.668002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.668002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"117'6177","reported_seq":6231,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.324770+0000","last_change":"2024-03-26T21:30:38.544003+0000","last_active":"2024-03-26T21:32:01.324770+0000","last_peered":"2024-03-26T21:32:01.324770+0000","last_clean":"2024-03-26T21:32:01.324770+0000","last_became_active":"2024-03-26T21:30:38.543645+0000","last_became_peered":"2024-03-26T21:30:38.543645+0000","last_unstale":"2024-03-26T21:32:01.324770+0000","last_undegraded":"2024-03-26T21:32:01.324770+0000","last_fullsized":"2024-03-26T21:32:01.324770+0000","mapping_epoch":95,"log_start":"102'1100","ondisk_log_start":"102'1100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404815872,"num_objects":6177,"num_object_clones":0,"num_object_copies":18531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":395328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688322+0000","last_change":"2024-03-26T21:30:12.503906+0000","last_active":"2024-03-26T21:31:59.688322+0000","last_peered":"2024-03-26T21:31:59.688322+0000","last_clean":"2024-03-26T21:31:59.688322+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:31:59.688322+0000","last_undegraded":"2024-03-26T21:31:59.688322+0000","last_fullsized":"2024-03-26T21:31:59.688322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.503821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.503821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":133,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.670464+0000","last_change":"2024-03-26T21:30:31.738782+0000","last_active":"2024-03-26T21:31:59.670464+0000","last_peered":"2024-03-26T21:31:59.670464+0000","last_clean":"2024-03-26T21:30:19.145088+0000","last_became_active":"2024-03-26T21:30:31.738782+0000","last_became_peered":"2024-03-26T21:30:31.738782+0000","last_unstale":"2024-03-26T21:31:59.670464+0000","last_undegraded":"2024-03-26T21:30:31.736341+0000","last_fullsized":"2024-03-26T21:30:31.736205+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.687862+0000","last_change":"2024-03-26T21:30:07.493378+0000","last_active":"2024-03-26T21:31:59.687862+0000","last_peered":"2024-03-26T21:31:59.687862+0000","last_clean":"2024-03-26T21:31:59.687862+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:31:59.687862+0000","last_undegraded":"2024-03-26T21:31:59.687862+0000","last_fullsized":"2024-03-26T21:31:59.687862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:07.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:07.493257+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":158,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.669523+0000","last_change":"2024-03-26T21:30:31.743316+0000","last_active":"2024-03-26T21:31:59.669523+0000","last_peered":"2024-03-26T21:31:59.669523+0000","last_clean":"2024-03-26T21:30:29.906838+0000","last_became_active":"2024-03-26T21:30:31.743316+0000","last_became_peered":"2024-03-26T21:30:31.743316+0000","last_unstale":"2024-03-26T21:31:59.669523+0000","last_undegraded":"2024-03-26T21:30:31.736202+0000","last_fullsized":"2024-03-26T21:30:31.736088+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"117'6146","reported_seq":6200,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.458479+0000","last_change":"2024-03-26T21:30:38.548354+0000","last_active":"2024-03-26T21:32:01.458479+0000","last_peered":"2024-03-26T21:32:01.458479+0000","last_clean":"2024-03-26T21:32:01.458479+0000","last_became_active":"2024-03-26T21:30:38.548054+0000","last_became_peered":"2024-03-26T21:30:38.548054+0000","last_unstale":"2024-03-26T21:32:01.458479+0000","last_undegraded":"2024-03-26T21:32:01.458479+0000","last_fullsized":"2024-03-26T21:32:01.458479+0000","mapping_epoch":95,"log_start":"102'1500","ondisk_log_start":"102'1500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":393344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681304+0000","last_change":"2024-03-26T21:30:40.780018+0000","last_active":"2024-03-26T21:31:59.681304+0000","last_peered":"2024-03-26T21:31:59.681304+0000","last_clean":"2024-03-26T21:31:59.681304+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:31:59.681304+0000","last_undegraded":"2024-03-26T21:31:59.681304+0000","last_fullsized":"2024-03-26T21:31:59.681304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:40.779910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:40.779910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:10.033861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":130,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.670404+0000","last_change":"2024-03-26T21:30:31.744165+0000","last_active":"2024-03-26T21:31:59.670404+0000","last_peered":"2024-03-26T21:31:59.670404+0000","last_clean":"2024-03-26T21:30:19.145058+0000","last_became_active":"2024-03-26T21:30:31.744165+0000","last_became_peered":"2024-03-26T21:30:31.744165+0000","last_unstale":"2024-03-26T21:31:59.670404+0000","last_undegraded":"2024-03-26T21:30:31.735909+0000","last_fullsized":"2024-03-26T21:30:31.735789+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":97,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.686696+0000","last_change":"2024-03-26T21:30:31.740300+0000","last_active":"2024-03-26T21:31:59.686696+0000","last_peered":"2024-03-26T21:31:59.686696+0000","last_clean":"2024-03-26T21:28:42.051440+0000","last_became_active":"2024-03-26T21:30:31.740300+0000","last_became_peered":"2024-03-26T21:30:31.740300+0000","last_unstale":"2024-03-26T21:31:59.686696+0000","last_undegraded":"2024-03-26T21:30:31.730730+0000","last_fullsized":"2024-03-26T21:30:31.730592+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.680168+0000","last_change":"2024-03-26T21:31:33.234806+0000","last_active":"2024-03-26T21:31:59.680168+0000","last_peered":"2024-03-26T21:31:59.680168+0000","last_clean":"2024-03-26T21:31:59.680168+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:31:59.680168+0000","last_undegraded":"2024-03-26T21:31:59.680168+0000","last_fullsized":"2024-03-26T21:31:59.680168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:33.234681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:33.234681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.001485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"117'6311","reported_seq":6362,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:01.401307+0000","last_change":"2024-03-26T21:30:53.719400+0000","last_active":"2024-03-26T21:32:01.401307+0000","last_peered":"2024-03-26T21:32:01.401307+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.546367+0000","last_became_peered":"2024-03-26T21:30:38.546367+0000","last_unstale":"2024-03-26T21:32:01.401307+0000","last_undegraded":"2024-03-26T21:30:53.709328+0000","last_fullsized":"2024-03-26T21:30:38.539071+0000","mapping_epoch":95,"log_start":"101'700","ondisk_log_start":"101'700","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.559137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413597696,"num_objects":6311,"num_object_clones":0,"num_object_copies":18933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":403904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6311}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.669092+0000","last_change":"2024-03-26T21:30:39.632228+0000","last_active":"2024-03-26T21:31:59.669092+0000","last_peered":"2024-03-26T21:31:59.669092+0000","last_clean":"2024-03-26T21:31:59.669092+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:31:59.669092+0000","last_undegraded":"2024-03-26T21:31:59.669092+0000","last_fullsized":"2024-03-26T21:31:59.669092+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:39.632108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:39.632108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679665+0000","last_change":"2024-03-26T21:31:30.180324+0000","last_active":"2024-03-26T21:31:59.679665+0000","last_peered":"2024-03-26T21:31:59.679665+0000","last_clean":"2024-03-26T21:31:59.679665+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:31:59.679665+0000","last_undegraded":"2024-03-26T21:31:59.679665+0000","last_fullsized":"2024-03-26T21:31:59.679665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:30.180213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.180213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.156578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.671609+0000","last_change":"2024-03-26T21:31:16.725440+0000","last_active":"2024-03-26T21:31:59.671609+0000","last_peered":"2024-03-26T21:31:59.671609+0000","last_clean":"2024-03-26T21:31:59.671609+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:31:59.671609+0000","last_undegraded":"2024-03-26T21:31:59.671609+0000","last_fullsized":"2024-03-26T21:31:59.671609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:16.725334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:16.725334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.801808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":164,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.675589+0000","last_change":"2024-03-26T21:31:28.640751+0000","last_active":"2024-03-26T21:31:59.675589+0000","last_peered":"2024-03-26T21:31:59.675589+0000","last_clean":"2024-03-26T21:31:59.675589+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:31:59.675589+0000","last_undegraded":"2024-03-26T21:31:59.675589+0000","last_fullsized":"2024-03-26T21:31:59.675589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:31:28.640572+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:31:28.640572+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.640572+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.382562+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"117'6266","reported_seq":6317,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:01.354207+0000","last_change":"2024-03-26T21:30:53.710423+0000","last_active":"2024-03-26T21:32:01.354207+0000","last_peered":"2024-03-26T21:32:01.354207+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.540186+0000","last_became_peered":"2024-03-26T21:30:38.540186+0000","last_unstale":"2024-03-26T21:32:01.354207+0000","last_undegraded":"2024-03-26T21:30:53.707983+0000","last_fullsized":"2024-03-26T21:30:38.534077+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.376781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410648576,"num_objects":6266,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":6266}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674055+0000","last_change":"2024-03-26T21:31:37.722079+0000","last_active":"2024-03-26T21:31:59.674055+0000","last_peered":"2024-03-26T21:31:59.674055+0000","last_clean":"2024-03-26T21:31:59.674055+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:31:59.674055+0000","last_undegraded":"2024-03-26T21:31:59.674055+0000","last_fullsized":"2024-03-26T21:31:59.674055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.721821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:57.948602+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.680369+0000","last_change":"2024-03-26T21:30:31.741930+0000","last_active":"2024-03-26T21:31:59.680369+0000","last_peered":"2024-03-26T21:31:59.680369+0000","last_clean":"2024-03-26T21:30:19.264875+0000","last_became_active":"2024-03-26T21:30:31.741930+0000","last_became_peered":"2024-03-26T21:30:31.741930+0000","last_unstale":"2024-03-26T21:31:59.680369+0000","last_undegraded":"2024-03-26T21:31:59.680369+0000","last_fullsized":"2024-03-26T21:30:31.734873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688627+0000","last_change":"2024-03-26T21:31:39.585713+0000","last_active":"2024-03-26T21:31:59.688627+0000","last_peered":"2024-03-26T21:31:59.688627+0000","last_clean":"2024-03-26T21:31:59.688627+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:31:59.688627+0000","last_undegraded":"2024-03-26T21:31:59.688627+0000","last_fullsized":"2024-03-26T21:31:59.688627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.585631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.992136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":127,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.674923+0000","last_change":"2024-03-26T21:30:31.747569+0000","last_active":"2024-03-26T21:31:59.674923+0000","last_peered":"2024-03-26T21:31:59.674923+0000","last_clean":"2024-03-26T21:30:19.056675+0000","last_became_active":"2024-03-26T21:30:31.747569+0000","last_became_peered":"2024-03-26T21:30:31.747569+0000","last_unstale":"2024-03-26T21:31:59.674923+0000","last_undegraded":"2024-03-26T21:30:31.738166+0000","last_fullsized":"2024-03-26T21:30:31.738045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"117'6296","reported_seq":6350,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.269768+0000","last_change":"2024-03-26T21:30:38.549130+0000","last_active":"2024-03-26T21:32:01.269768+0000","last_peered":"2024-03-26T21:32:01.269768+0000","last_clean":"2024-03-26T21:32:01.269768+0000","last_became_active":"2024-03-26T21:30:38.548880+0000","last_became_peered":"2024-03-26T21:30:38.548880+0000","last_unstale":"2024-03-26T21:32:01.269768+0000","last_undegraded":"2024-03-26T21:32:01.269768+0000","last_fullsized":"2024-03-26T21:32:01.269768+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.792954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412614656,"num_objects":6296,"num_object_clones":0,"num_object_copies":18888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":402944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.684771+0000","last_change":"2024-03-26T21:30:31.738178+0000","last_active":"2024-03-26T21:31:59.684771+0000","last_peered":"2024-03-26T21:31:59.684771+0000","last_clean":"2024-03-26T21:30:19.792003+0000","last_became_active":"2024-03-26T21:30:31.738178+0000","last_became_peered":"2024-03-26T21:30:31.738178+0000","last_unstale":"2024-03-26T21:31:59.684771+0000","last_undegraded":"2024-03-26T21:31:59.684771+0000","last_fullsized":"2024-03-26T21:30:31.731810+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688159+0000","last_change":"2024-03-26T21:30:06.496381+0000","last_active":"2024-03-26T21:31:59.688159+0000","last_peered":"2024-03-26T21:31:59.688159+0000","last_clean":"2024-03-26T21:31:59.688159+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:31:59.688159+0000","last_undegraded":"2024-03-26T21:31:59.688159+0000","last_fullsized":"2024-03-26T21:31:59.688159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:30:06.496257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.496257+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":56,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.674579+0000","last_change":"2024-03-26T21:30:31.740606+0000","last_active":"2024-03-26T21:31:59.674579+0000","last_peered":"2024-03-26T21:31:59.674579+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740606+0000","last_became_peered":"2024-03-26T21:30:31.740606+0000","last_unstale":"2024-03-26T21:31:59.674579+0000","last_undegraded":"2024-03-26T21:31:59.674579+0000","last_fullsized":"2024-03-26T21:30:31.737530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:15.290476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:15.290476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":168,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688318+0000","last_change":"2024-03-26T21:31:30.659923+0000","last_active":"2024-03-26T21:31:59.688318+0000","last_peered":"2024-03-26T21:31:59.688318+0000","last_clean":"2024-03-26T21:31:59.688318+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:31:59.688318+0000","last_undegraded":"2024-03-26T21:31:59.688318+0000","last_fullsized":"2024-03-26T21:31:59.688318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:31:30.659832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.659832+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.256066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.675692+0000","last_change":"2024-03-26T21:30:31.745830+0000","last_active":"2024-03-26T21:31:59.675692+0000","last_peered":"2024-03-26T21:31:59.675692+0000","last_clean":"2024-03-26T21:30:19.056424+0000","last_became_active":"2024-03-26T21:30:31.745830+0000","last_became_peered":"2024-03-26T21:30:31.745830+0000","last_unstale":"2024-03-26T21:31:59.675692+0000","last_undegraded":"2024-03-26T21:31:59.675692+0000","last_fullsized":"2024-03-26T21:30:31.736422+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.721880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.721880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"117'6169","reported_seq":6219,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:01.415379+0000","last_change":"2024-03-26T21:30:53.720906+0000","last_active":"2024-03-26T21:32:01.415379+0000","last_peered":"2024-03-26T21:32:01.415379+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.544117+0000","last_became_peered":"2024-03-26T21:30:38.544117+0000","last_unstale":"2024-03-26T21:32:01.415379+0000","last_undegraded":"2024-03-26T21:30:53.719381+0000","last_fullsized":"2024-03-26T21:30:38.538732+0000","mapping_epoch":95,"log_start":"101'600","ondisk_log_start":"101'600","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.559123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404291584,"num_objects":6169,"num_object_clones":0,"num_object_copies":18507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6169,"num_write_kb":394816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.671663+0000","last_change":"2024-03-26T21:31:36.751872+0000","last_active":"2024-03-26T21:31:59.671663+0000","last_peered":"2024-03-26T21:31:59.671663+0000","last_clean":"2024-03-26T21:31:59.671663+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:31:59.671663+0000","last_undegraded":"2024-03-26T21:31:59.671663+0000","last_fullsized":"2024-03-26T21:31:59.671663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:31:36.751794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.345858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.685722+0000","last_change":"2024-03-26T21:31:23.146185+0000","last_active":"2024-03-26T21:31:59.685722+0000","last_peered":"2024-03-26T21:31:59.685722+0000","last_clean":"2024-03-26T21:31:59.685722+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:31:59.685722+0000","last_undegraded":"2024-03-26T21:31:59.685722+0000","last_fullsized":"2024-03-26T21:31:59.685722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:23.145963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:23.145963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:24.114030+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679560+0000","last_change":"2024-03-26T21:31:28.109942+0000","last_active":"2024-03-26T21:31:59.679560+0000","last_peered":"2024-03-26T21:31:59.679560+0000","last_clean":"2024-03-26T21:31:59.679560+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:31:59.679560+0000","last_undegraded":"2024-03-26T21:31:59.679560+0000","last_fullsized":"2024-03-26T21:31:59.679560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.109786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.109786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:46.736255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":156,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688159+0000","last_change":"2024-03-26T21:31:28.105503+0000","last_active":"2024-03-26T21:31:59.688159+0000","last_peered":"2024-03-26T21:31:59.688159+0000","last_clean":"2024-03-26T21:31:59.688159+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:31:59.688159+0000","last_undegraded":"2024-03-26T21:31:59.688159+0000","last_fullsized":"2024-03-26T21:31:59.688159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:31:28.105376+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.105376+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:48.590370+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.678247+0000","last_change":"2024-03-26T21:31:25.089539+0000","last_active":"2024-03-26T21:31:59.678247+0000","last_peered":"2024-03-26T21:31:59.678247+0000","last_clean":"2024-03-26T21:31:59.678247+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:31:59.678247+0000","last_undegraded":"2024-03-26T21:31:59.678247+0000","last_fullsized":"2024-03-26T21:31:59.678247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:25.089066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.089066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:52.881598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"117'6165","reported_seq":6217,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.400529+0000","last_change":"2024-03-26T21:30:38.551063+0000","last_active":"2024-03-26T21:32:01.400529+0000","last_peered":"2024-03-26T21:32:01.400529+0000","last_clean":"2024-03-26T21:32:01.400529+0000","last_became_active":"2024-03-26T21:30:38.550658+0000","last_became_peered":"2024-03-26T21:30:38.550658+0000","last_unstale":"2024-03-26T21:32:01.400529+0000","last_undegraded":"2024-03-26T21:32:01.400529+0000","last_fullsized":"2024-03-26T21:32:01.400529+0000","mapping_epoch":95,"log_start":"101'600","ondisk_log_start":"101'600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.559059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404029440,"num_objects":6165,"num_object_clones":0,"num_object_copies":18495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":394560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.688575+0000","last_change":"2024-03-26T21:30:31.744917+0000","last_active":"2024-03-26T21:31:59.688575+0000","last_peered":"2024-03-26T21:31:59.688575+0000","last_clean":"2024-03-26T21:30:19.907127+0000","last_became_active":"2024-03-26T21:30:31.744917+0000","last_became_peered":"2024-03-26T21:30:31.744917+0000","last_unstale":"2024-03-26T21:31:59.688575+0000","last_undegraded":"2024-03-26T21:31:59.688575+0000","last_fullsized":"2024-03-26T21:30:31.734562+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.472985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.472985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":372,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.669297+0000","last_change":"2024-03-26T21:30:32.576223+0000","last_active":"2024-03-26T21:31:59.669297+0000","last_peered":"2024-03-26T21:31:59.669297+0000","last_clean":"2024-03-26T21:31:59.669297+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:31:59.669297+0000","last_undegraded":"2024-03-26T21:31:59.669297+0000","last_fullsized":"2024-03-26T21:31:59.669297+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:30:32.576135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:30:32.576135+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679996+0000","last_change":"2024-03-26T21:31:14.873351+0000","last_active":"2024-03-26T21:31:59.679996+0000","last_peered":"2024-03-26T21:31:59.679996+0000","last_clean":"2024-03-26T21:31:59.679996+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:31:59.679996+0000","last_undegraded":"2024-03-26T21:31:59.679996+0000","last_fullsized":"2024-03-26T21:31:59.679996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:14.872965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.872965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.836988+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"94'25","reported_seq":192,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.676751+0000","last_change":"2024-03-26T21:30:31.737537+0000","last_active":"2024-03-26T21:31:59.676751+0000","last_peered":"2024-03-26T21:31:59.676751+0000","last_clean":"2024-03-26T21:30:29.907049+0000","last_became_active":"2024-03-26T21:30:31.737537+0000","last_became_peered":"2024-03-26T21:30:31.737537+0000","last_unstale":"2024-03-26T21:31:59.676751+0000","last_undegraded":"2024-03-26T21:30:31.733102+0000","last_fullsized":"2024-03-26T21:30:31.732979+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.670796+0000","last_change":"2024-03-26T21:31:15.753021+0000","last_active":"2024-03-26T21:31:59.670796+0000","last_peered":"2024-03-26T21:31:59.670796+0000","last_clean":"2024-03-26T21:31:59.670796+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:31:59.670796+0000","last_undegraded":"2024-03-26T21:31:59.670796+0000","last_fullsized":"2024-03-26T21:31:59.670796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:15.752907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:15.752907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:20.838766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"117'6183","reported_seq":6237,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.273131+0000","last_change":"2024-03-26T21:30:38.543845+0000","last_active":"2024-03-26T21:32:01.273131+0000","last_peered":"2024-03-26T21:32:01.273131+0000","last_clean":"2024-03-26T21:32:01.273131+0000","last_became_active":"2024-03-26T21:30:38.543531+0000","last_became_peered":"2024-03-26T21:30:38.543531+0000","last_unstale":"2024-03-26T21:32:01.273131+0000","last_undegraded":"2024-03-26T21:32:01.273131+0000","last_fullsized":"2024-03-26T21:32:01.273131+0000","mapping_epoch":95,"log_start":"102'1000","ondisk_log_start":"102'1000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.728142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405209088,"num_objects":6183,"num_object_clones":0,"num_object_copies":18549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":395712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.685682+0000","last_change":"2024-03-26T21:31:39.079125+0000","last_active":"2024-03-26T21:31:59.685682+0000","last_peered":"2024-03-26T21:31:59.685682+0000","last_clean":"2024-03-26T21:31:59.685682+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:31:59.685682+0000","last_undegraded":"2024-03-26T21:31:59.685682+0000","last_fullsized":"2024-03-26T21:31:59.685682+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.079005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.079005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:03.373633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.674371+0000","last_change":"2024-03-26T21:30:31.743896+0000","last_active":"2024-03-26T21:31:59.674371+0000","last_peered":"2024-03-26T21:31:59.674371+0000","last_clean":"2024-03-26T21:30:19.055870+0000","last_became_active":"2024-03-26T21:30:31.743896+0000","last_became_peered":"2024-03-26T21:30:31.743896+0000","last_unstale":"2024-03-26T21:31:59.674371+0000","last_undegraded":"2024-03-26T21:31:59.674371+0000","last_fullsized":"2024-03-26T21:30:31.733630+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:30:17.708355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":338,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.669241+0000","last_change":"2024-03-26T21:31:29.792073+0000","last_active":"2024-03-26T21:31:59.669241+0000","last_peered":"2024-03-26T21:31:59.669241+0000","last_clean":"2024-03-26T21:31:59.669241+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:31:59.669241+0000","last_undegraded":"2024-03-26T21:31:59.669241+0000","last_fullsized":"2024-03-26T21:31:59.669241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:31:29.791964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.791964+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:47.167445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":200,"num_read_kb":195,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":162,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.686689+0000","last_change":"2024-03-26T21:31:23.770810+0000","last_active":"2024-03-26T21:31:59.686689+0000","last_peered":"2024-03-26T21:31:59.686689+0000","last_clean":"2024-03-26T21:31:59.686689+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:31:59.686689+0000","last_undegraded":"2024-03-26T21:31:59.686689+0000","last_fullsized":"2024-03-26T21:31:59.686689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:31:23.770682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:23.770682+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:37.819857+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.689196+0000","last_change":"2024-03-26T21:30:37.633283+0000","last_active":"2024-03-26T21:31:59.689196+0000","last_peered":"2024-03-26T21:31:59.689196+0000","last_clean":"2024-03-26T21:31:59.689196+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:31:59.689196+0000","last_undegraded":"2024-03-26T21:31:59.689196+0000","last_fullsized":"2024-03-26T21:31:59.689196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.633179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.633179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"117'6245","reported_seq":6299,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.300968+0000","last_change":"2024-03-26T21:30:38.544124+0000","last_active":"2024-03-26T21:32:01.300968+0000","last_peered":"2024-03-26T21:32:01.300968+0000","last_clean":"2024-03-26T21:32:01.300968+0000","last_became_active":"2024-03-26T21:30:38.543747+0000","last_became_peered":"2024-03-26T21:30:38.543747+0000","last_unstale":"2024-03-26T21:32:01.300968+0000","last_undegraded":"2024-03-26T21:32:01.300968+0000","last_fullsized":"2024-03-26T21:32:01.300968+0000","mapping_epoch":95,"log_start":"101'500","ondisk_log_start":"101'500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.549257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409272320,"num_objects":6245,"num_object_clones":0,"num_object_copies":18735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":399680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688347+0000","last_change":"2024-03-26T21:31:30.064302+0000","last_active":"2024-03-26T21:31:59.688347+0000","last_peered":"2024-03-26T21:31:59.688347+0000","last_clean":"2024-03-26T21:31:59.688347+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:31:59.688347+0000","last_undegraded":"2024-03-26T21:31:59.688347+0000","last_fullsized":"2024-03-26T21:31:59.688347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:30.064009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.064009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.180293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.685903+0000","last_change":"2024-03-26T21:31:25.137611+0000","last_active":"2024-03-26T21:31:59.685903+0000","last_peered":"2024-03-26T21:31:59.685903+0000","last_clean":"2024-03-26T21:31:59.685903+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:31:59.685903+0000","last_undegraded":"2024-03-26T21:31:59.685903+0000","last_fullsized":"2024-03-26T21:31:59.685903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:25.137255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.137255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.125501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":160,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674768+0000","last_change":"2024-03-26T21:31:32.754170+0000","last_active":"2024-03-26T21:31:59.674768+0000","last_peered":"2024-03-26T21:31:59.674768+0000","last_clean":"2024-03-26T21:31:59.674768+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:31:59.674768+0000","last_undegraded":"2024-03-26T21:31:59.674768+0000","last_fullsized":"2024-03-26T21:31:59.674768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:31:32.754010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.754010+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.783769+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.675683+0000","last_change":"2024-03-26T21:31:21.554718+0000","last_active":"2024-03-26T21:31:59.675683+0000","last_peered":"2024-03-26T21:31:59.675683+0000","last_clean":"2024-03-26T21:31:59.675683+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:31:59.675683+0000","last_undegraded":"2024-03-26T21:31:59.675683+0000","last_fullsized":"2024-03-26T21:31:59.675683+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:21.554604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:21.554604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:40.440766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"117'6067","reported_seq":6118,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:01.479696+0000","last_change":"2024-03-26T21:30:53.715187+0000","last_active":"2024-03-26T21:32:01.479696+0000","last_peered":"2024-03-26T21:32:01.479696+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.547364+0000","last_became_peered":"2024-03-26T21:30:38.547364+0000","last_unstale":"2024-03-26T21:32:01.479696+0000","last_undegraded":"2024-03-26T21:30:53.713329+0000","last_fullsized":"2024-03-26T21:30:38.537882+0000","mapping_epoch":95,"log_start":"102'1400","ondisk_log_start":"102'1400","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.381586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397606912,"num_objects":6067,"num_object_clones":0,"num_object_copies":18201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":388288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":6067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.673835+0000","last_change":"2024-03-26T21:30:12.703236+0000","last_active":"2024-03-26T21:31:59.673835+0000","last_peered":"2024-03-26T21:31:59.673835+0000","last_clean":"2024-03-26T21:31:59.673835+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:31:59.673835+0000","last_undegraded":"2024-03-26T21:31:59.673835+0000","last_fullsized":"2024-03-26T21:31:59.673835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:12.703154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:12.703154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688961+0000","last_change":"2024-03-26T21:30:38.590322+0000","last_active":"2024-03-26T21:31:59.688961+0000","last_peered":"2024-03-26T21:31:59.688961+0000","last_clean":"2024-03-26T21:31:59.688961+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:31:59.688961+0000","last_undegraded":"2024-03-26T21:31:59.688961+0000","last_fullsized":"2024-03-26T21:31:59.688961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:38.590254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:38.590254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:07.324127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":134,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.687070+0000","last_change":"2024-03-26T21:30:31.739773+0000","last_active":"2024-03-26T21:31:59.687070+0000","last_peered":"2024-03-26T21:31:59.687070+0000","last_clean":"2024-03-26T21:30:19.056603+0000","last_became_active":"2024-03-26T21:30:31.739773+0000","last_became_peered":"2024-03-26T21:30:31.739773+0000","last_unstale":"2024-03-26T21:31:59.687070+0000","last_undegraded":"2024-03-26T21:30:31.733621+0000","last_fullsized":"2024-03-26T21:30:31.733509+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:02.076064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.076064+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":162,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.674578+0000","last_change":"2024-03-26T21:30:11.661994+0000","last_active":"2024-03-26T21:31:59.674578+0000","last_peered":"2024-03-26T21:31:59.674578+0000","last_clean":"2024-03-26T21:31:59.674578+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:31:59.674578+0000","last_undegraded":"2024-03-26T21:31:59.674578+0000","last_fullsized":"2024-03-26T21:31:59.674578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.661905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.661905+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"117'6178","reported_seq":6230,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.506363+0000","last_change":"2024-03-26T21:30:38.550210+0000","last_active":"2024-03-26T21:32:01.506363+0000","last_peered":"2024-03-26T21:32:01.506363+0000","last_clean":"2024-03-26T21:32:01.506363+0000","last_became_active":"2024-03-26T21:30:38.549806+0000","last_became_peered":"2024-03-26T21:30:38.549806+0000","last_unstale":"2024-03-26T21:32:01.506363+0000","last_undegraded":"2024-03-26T21:32:01.506363+0000","last_fullsized":"2024-03-26T21:32:01.506363+0000","mapping_epoch":95,"log_start":"102'1500","ondisk_log_start":"102'1500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:04.813813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404881408,"num_objects":6178,"num_object_clones":0,"num_object_copies":18534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6178,"num_write_kb":395392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.670046+0000","last_change":"2024-03-26T21:30:38.599138+0000","last_active":"2024-03-26T21:31:59.670046+0000","last_peered":"2024-03-26T21:31:59.670046+0000","last_clean":"2024-03-26T21:31:59.670046+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:31:59.670046+0000","last_undegraded":"2024-03-26T21:31:59.670046+0000","last_fullsized":"2024-03-26T21:31:59.670046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:38.599066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:38.599066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.801185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":116,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.686790+0000","last_change":"2024-03-26T21:30:31.746821+0000","last_active":"2024-03-26T21:31:59.686790+0000","last_peered":"2024-03-26T21:31:59.686790+0000","last_clean":"2024-03-26T21:30:19.056856+0000","last_became_active":"2024-03-26T21:30:31.746821+0000","last_became_peered":"2024-03-26T21:30:31.746821+0000","last_unstale":"2024-03-26T21:31:59.686790+0000","last_undegraded":"2024-03-26T21:30:31.736673+0000","last_fullsized":"2024-03-26T21:30:31.736528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.552215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":95,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.687856+0000","last_change":"2024-03-26T21:30:31.739394+0000","last_active":"2024-03-26T21:31:59.687856+0000","last_peered":"2024-03-26T21:31:59.687856+0000","last_clean":"2024-03-26T21:28:42.253768+0000","last_became_active":"2024-03-26T21:30:31.739394+0000","last_became_peered":"2024-03-26T21:30:31.739394+0000","last_unstale":"2024-03-26T21:31:59.687856+0000","last_undegraded":"2024-03-26T21:30:31.732454+0000","last_fullsized":"2024-03-26T21:30:31.732332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"94'9","reported_seq":166,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.679333+0000","last_change":"2024-03-26T21:30:31.745653+0000","last_active":"2024-03-26T21:31:59.679333+0000","last_peered":"2024-03-26T21:31:59.679333+0000","last_clean":"2024-03-26T21:30:29.906563+0000","last_became_active":"2024-03-26T21:30:31.745653+0000","last_became_peered":"2024-03-26T21:30:31.745653+0000","last_unstale":"2024-03-26T21:31:59.679333+0000","last_undegraded":"2024-03-26T21:30:31.730030+0000","last_fullsized":"2024-03-26T21:30:31.729924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:29:59.694434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:59.694434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"117'6196","reported_seq":6247,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:01.464653+0000","last_change":"2024-03-26T21:30:53.715233+0000","last_active":"2024-03-26T21:32:01.464653+0000","last_peered":"2024-03-26T21:32:01.464653+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.548505+0000","last_became_peered":"2024-03-26T21:30:38.548505+0000","last_unstale":"2024-03-26T21:32:01.464653+0000","last_undegraded":"2024-03-26T21:30:53.710995+0000","last_fullsized":"2024-03-26T21:30:38.537548+0000","mapping_epoch":95,"log_start":"102'1500","ondisk_log_start":"102'1500","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:06.081812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406061056,"num_objects":6196,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":396544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":6196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.682001+0000","last_change":"2024-03-26T21:30:31.738939+0000","last_active":"2024-03-26T21:31:59.682001+0000","last_peered":"2024-03-26T21:31:59.682001+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738939+0000","last_became_peered":"2024-03-26T21:30:31.738939+0000","last_unstale":"2024-03-26T21:31:59.682001+0000","last_undegraded":"2024-03-26T21:31:59.682001+0000","last_fullsized":"2024-03-26T21:30:31.730286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679795+0000","last_change":"2024-03-26T21:29:56.690968+0000","last_active":"2024-03-26T21:31:59.679795+0000","last_peered":"2024-03-26T21:31:59.679795+0000","last_clean":"2024-03-26T21:31:59.679795+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:31:59.679795+0000","last_undegraded":"2024-03-26T21:31:59.679795+0000","last_fullsized":"2024-03-26T21:31:59.679795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:56.690884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:56.690884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":148,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679117+0000","last_change":"2024-03-26T21:30:35.771996+0000","last_active":"2024-03-26T21:31:59.679117+0000","last_peered":"2024-03-26T21:31:59.679117+0000","last_clean":"2024-03-26T21:31:59.679117+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:31:59.679117+0000","last_undegraded":"2024-03-26T21:31:59.679117+0000","last_fullsized":"2024-03-26T21:31:59.679117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:30:35.771902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:35.771902+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:02.925148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.680375+0000","last_change":"2024-03-26T21:30:31.738110+0000","last_active":"2024-03-26T21:31:59.680375+0000","last_peered":"2024-03-26T21:31:59.680375+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.738110+0000","last_became_peered":"2024-03-26T21:30:31.738110+0000","last_unstale":"2024-03-26T21:31:59.680375+0000","last_undegraded":"2024-03-26T21:31:59.680375+0000","last_fullsized":"2024-03-26T21:30:31.729956+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"117'6199","reported_seq":6253,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.284661+0000","last_change":"2024-03-26T21:30:38.540136+0000","last_active":"2024-03-26T21:32:01.284661+0000","last_peered":"2024-03-26T21:32:01.284661+0000","last_clean":"2024-03-26T21:32:01.284661+0000","last_became_active":"2024-03-26T21:30:38.539838+0000","last_became_peered":"2024-03-26T21:30:38.539838+0000","last_unstale":"2024-03-26T21:32:01.284661+0000","last_undegraded":"2024-03-26T21:32:01.284661+0000","last_fullsized":"2024-03-26T21:32:01.284661+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406257664,"num_objects":6199,"num_object_clones":0,"num_object_copies":18597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":396736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.688335+0000","last_change":"2024-03-26T21:31:34.602108+0000","last_active":"2024-03-26T21:31:59.688335+0000","last_peered":"2024-03-26T21:31:59.688335+0000","last_clean":"2024-03-26T21:31:59.688335+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:31:59.688335+0000","last_undegraded":"2024-03-26T21:31:59.688335+0000","last_fullsized":"2024-03-26T21:31:59.688335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:34.602001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:34.602001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:59.757795+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"94'9","reported_seq":99,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.688608+0000","last_change":"2024-03-26T21:30:31.738518+0000","last_active":"2024-03-26T21:31:59.688608+0000","last_peered":"2024-03-26T21:31:59.688608+0000","last_clean":"2024-03-26T21:28:42.046048+0000","last_became_active":"2024-03-26T21:30:31.738518+0000","last_became_peered":"2024-03-26T21:30:31.738518+0000","last_unstale":"2024-03-26T21:31:59.688608+0000","last_undegraded":"2024-03-26T21:30:31.734182+0000","last_fullsized":"2024-03-26T21:30:31.734060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.407216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.407216+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679030+0000","last_change":"2024-03-26T21:31:29.955815+0000","last_active":"2024-03-26T21:31:59.679030+0000","last_peered":"2024-03-26T21:31:59.679030+0000","last_clean":"2024-03-26T21:31:59.679030+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:31:59.679030+0000","last_undegraded":"2024-03-26T21:31:59.679030+0000","last_fullsized":"2024-03-26T21:31:59.679030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:51.604999+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":144,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.688319+0000","last_change":"2024-03-26T21:30:31.746320+0000","last_active":"2024-03-26T21:31:59.688319+0000","last_peered":"2024-03-26T21:31:59.688319+0000","last_clean":"2024-03-26T21:30:19.057783+0000","last_became_active":"2024-03-26T21:30:31.746320+0000","last_became_peered":"2024-03-26T21:30:31.746320+0000","last_unstale":"2024-03-26T21:31:59.688319+0000","last_undegraded":"2024-03-26T21:30:31.733035+0000","last_fullsized":"2024-03-26T21:30:31.732940+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:11.086876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.086876+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"117'6124","reported_seq":6178,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.281852+0000","last_change":"2024-03-26T21:30:38.544298+0000","last_active":"2024-03-26T21:32:01.281852+0000","last_peered":"2024-03-26T21:32:01.281852+0000","last_clean":"2024-03-26T21:32:01.281852+0000","last_became_active":"2024-03-26T21:30:38.543981+0000","last_became_peered":"2024-03-26T21:30:38.543981+0000","last_unstale":"2024-03-26T21:32:01.281852+0000","last_undegraded":"2024-03-26T21:32:01.281852+0000","last_fullsized":"2024-03-26T21:32:01.281852+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:05.792972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401342464,"num_objects":6124,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":391936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.681061+0000","last_change":"2024-03-26T21:31:34.924180+0000","last_active":"2024-03-26T21:31:59.681061+0000","last_peered":"2024-03-26T21:31:59.681061+0000","last_clean":"2024-03-26T21:31:59.681061+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:31:59.681061+0000","last_undegraded":"2024-03-26T21:31:59.681061+0000","last_fullsized":"2024-03-26T21:31:59.681061+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:31:34.924056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.555962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":107,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.689001+0000","last_change":"2024-03-26T21:30:31.742736+0000","last_active":"2024-03-26T21:31:59.689001+0000","last_peered":"2024-03-26T21:31:59.689001+0000","last_clean":"2024-03-26T21:30:19.907576+0000","last_became_active":"2024-03-26T21:30:31.742736+0000","last_became_peered":"2024-03-26T21:30:31.742736+0000","last_unstale":"2024-03-26T21:31:59.689001+0000","last_undegraded":"2024-03-26T21:31:59.689001+0000","last_fullsized":"2024-03-26T21:30:31.735097+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.480811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":56,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.679367+0000","last_change":"2024-03-26T21:30:31.738606+0000","last_active":"2024-03-26T21:31:59.679367+0000","last_peered":"2024-03-26T21:31:59.679367+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.738606+0000","last_became_peered":"2024-03-26T21:30:31.738606+0000","last_unstale":"2024-03-26T21:31:59.679367+0000","last_undegraded":"2024-03-26T21:31:59.679367+0000","last_fullsized":"2024-03-26T21:30:31.734264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":93,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.676732+0000","last_change":"2024-03-26T21:30:31.741438+0000","last_active":"2024-03-26T21:31:59.676732+0000","last_peered":"2024-03-26T21:31:59.676732+0000","last_clean":"2024-03-26T21:28:42.249232+0000","last_became_active":"2024-03-26T21:30:31.741438+0000","last_became_peered":"2024-03-26T21:30:31.741438+0000","last_unstale":"2024-03-26T21:31:59.676732+0000","last_undegraded":"2024-03-26T21:30:31.733158+0000","last_fullsized":"2024-03-26T21:30:31.733038+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"117'6266","reported_seq":6320,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.359509+0000","last_change":"2024-03-26T21:30:38.545760+0000","last_active":"2024-03-26T21:32:01.359509+0000","last_peered":"2024-03-26T21:32:01.359509+0000","last_clean":"2024-03-26T21:32:01.359509+0000","last_became_active":"2024-03-26T21:30:38.545485+0000","last_became_peered":"2024-03-26T21:30:38.545485+0000","last_unstale":"2024-03-26T21:32:01.359509+0000","last_undegraded":"2024-03-26T21:32:01.359509+0000","last_fullsized":"2024-03-26T21:32:01.359509+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410648576,"num_objects":6266,"num_object_clones":0,"num_object_copies":18798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.679165+0000","last_change":"2024-03-26T21:30:31.741781+0000","last_active":"2024-03-26T21:31:59.679165+0000","last_peered":"2024-03-26T21:31:59.679165+0000","last_clean":"2024-03-26T21:30:19.265083+0000","last_became_active":"2024-03-26T21:30:31.741781+0000","last_became_peered":"2024-03-26T21:30:31.741781+0000","last_unstale":"2024-03-26T21:31:59.679165+0000","last_undegraded":"2024-03-26T21:31:59.679165+0000","last_fullsized":"2024-03-26T21:30:31.735785+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.923329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.686070+0000","last_change":"2024-03-26T21:30:31.743030+0000","last_active":"2024-03-26T21:31:59.686070+0000","last_peered":"2024-03-26T21:31:59.686070+0000","last_clean":"2024-03-26T21:30:19.057039+0000","last_became_active":"2024-03-26T21:30:31.743030+0000","last_became_peered":"2024-03-26T21:30:31.743030+0000","last_unstale":"2024-03-26T21:31:59.686070+0000","last_undegraded":"2024-03-26T21:31:59.686070+0000","last_fullsized":"2024-03-26T21:30:31.730593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.674968+0000","last_change":"2024-03-26T21:30:31.741495+0000","last_active":"2024-03-26T21:31:59.674968+0000","last_peered":"2024-03-26T21:31:59.674968+0000","last_clean":"2024-03-26T21:30:19.906082+0000","last_became_active":"2024-03-26T21:30:31.741495+0000","last_became_peered":"2024-03-26T21:30:31.741495+0000","last_unstale":"2024-03-26T21:31:59.674968+0000","last_undegraded":"2024-03-26T21:31:59.674968+0000","last_fullsized":"2024-03-26T21:30:31.732617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":102,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.680061+0000","last_change":"2024-03-26T21:30:31.741186+0000","last_active":"2024-03-26T21:31:59.680061+0000","last_peered":"2024-03-26T21:31:59.680061+0000","last_clean":"2024-03-26T21:28:42.246719+0000","last_became_active":"2024-03-26T21:30:31.741186+0000","last_became_peered":"2024-03-26T21:30:31.741186+0000","last_unstale":"2024-03-26T21:31:59.680061+0000","last_undegraded":"2024-03-26T21:30:31.735163+0000","last_fullsized":"2024-03-26T21:30:31.735041+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:30:19.267430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:19.267430+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"117'6146","reported_seq":6200,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:32:01.409087+0000","last_change":"2024-03-26T21:30:38.551085+0000","last_active":"2024-03-26T21:32:01.409087+0000","last_peered":"2024-03-26T21:32:01.409087+0000","last_clean":"2024-03-26T21:32:01.409087+0000","last_became_active":"2024-03-26T21:30:38.550667+0000","last_became_peered":"2024-03-26T21:30:38.550667+0000","last_unstale":"2024-03-26T21:32:01.409087+0000","last_undegraded":"2024-03-26T21:32:01.409087+0000","last_fullsized":"2024-03-26T21:32:01.409087+0000","mapping_epoch":95,"log_start":"101'500","ondisk_log_start":"101'500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:03.559107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":393344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.685609+0000","last_change":"2024-03-26T21:30:31.739518+0000","last_active":"2024-03-26T21:31:59.685609+0000","last_peered":"2024-03-26T21:31:59.685609+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.739518+0000","last_became_peered":"2024-03-26T21:30:31.739518+0000","last_unstale":"2024-03-26T21:31:59.685609+0000","last_undegraded":"2024-03-26T21:31:59.685609+0000","last_fullsized":"2024-03-26T21:30:31.730680+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":170,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-26T21:31:59.679864+0000","last_change":"2024-03-26T21:31:29.192047+0000","last_active":"2024-03-26T21:31:59.679864+0000","last_peered":"2024-03-26T21:31:59.679864+0000","last_clean":"2024-03-26T21:31:59.679864+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:31:59.679864+0000","last_undegraded":"2024-03-26T21:31:59.679864+0000","last_fullsized":"2024-03-26T21:31:59.679864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:31:29.191678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.191678+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.551761+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":115,"reported_epoch":117,"state":"active+undersized","last_fresh":"2024-03-26T21:31:59.680627+0000","last_change":"2024-03-26T21:30:31.743458+0000","last_active":"2024-03-26T21:31:59.680627+0000","last_peered":"2024-03-26T21:31:59.680627+0000","last_clean":"2024-03-26T21:30:19.791720+0000","last_became_active":"2024-03-26T21:30:31.743458+0000","last_became_peered":"2024-03-26T21:30:31.743458+0000","last_unstale":"2024-03-26T21:31:59.680627+0000","last_undegraded":"2024-03-26T21:31:59.680627+0000","last_fullsized":"2024-03-26T21:30:31.731249+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":120,"reported_epoch":117,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:31:59.681386+0000","last_change":"2024-03-26T21:30:31.741371+0000","last_active":"2024-03-26T21:31:59.681386+0000","last_peered":"2024-03-26T21:31:59.681386+0000","last_clean":"2024-03-26T21:29:37.643762+0000","last_became_active":"2024-03-26T21:30:31.741371+0000","last_became_peered":"2024-03-26T21:30:31.741371+0000","last_unstale":"2024-03-26T21:31:59.681386+0000","last_undegraded":"2024-03-26T21:30:31.731807+0000","last_fullsized":"2024-03-26T21:30:31.731688+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:01.081184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6496845824,"num_objects":99134,"num_object_clones":0,"num_object_copies":297402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99134,"num_write_kb":6344576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17457479680,"data_stored":17457479680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88234,"ondisk_log_size":88234,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":195,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":81,"acting":81,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042588,"kb_used_data":1971156,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91702564,"statfs":{"total":95995035648,"available":93903425536,"internally_reserved":0,"allocated":2018463744,"data_stored":2017985722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364852,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2037056,"kb_used_data":1965620,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91708096,"statfs":{"total":95995035648,"available":93909090304,"internally_reserved":0,"allocated":2012794880,"data_stored":2012347214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2436460,"kb_used_data":2373348,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91308692,"statfs":{"total":95995035648,"available":93500100608,"internally_reserved":0,"allocated":2430308352,"data_stored":2429812853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3269088,"kb_used_data":3157396,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90476064,"statfs":{"total":95995035648,"available":92647489536,"internally_reserved":0,"allocated":3233173504,"data_stored":3232694788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2440636,"kb_used_data":2360876,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":91304516,"statfs":{"total":95995035648,"available":93495824384,"internally_reserved":0,"allocated":2417537024,"data_stored":2417097121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3652388,"kb_used_data":3549016,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90092764,"statfs":{"total":95995035648,"available":92254990336,"internally_reserved":0,"allocated":3634192384,"data_stored":3633708214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27632,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717520,"statfs":{"total":95995035648,"available":95966740480,"internally_reserved":0,"allocated":630784,"data_stored":285906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659191,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621900,"kb_used_data":1576836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92123252,"statfs":{"total":95995035648,"available":94334210048,"internally_reserved":0,"allocated":1614680064,"data_stored":1614216154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3256025088,"data_stored":3256025088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2437218304,"data_stored":2437218304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2439249920,"data_stored":2439249920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3649568768,"data_stored":3649568768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1626341376,"data_stored":1626341376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2021457920,"data_stored":2021457920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2027618304,"data_stored":2027618304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:32:06.787 INFO:tasks.thrashosds.thrasher:Setting 5.1a to [5, 6, 0] 2024-03-26T21:32:06.788 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1a', '5', '6', '0'] 2024-03-26T21:32:06.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 5.1a 5 6 0 2024-03-26T21:32:06.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:06.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:06.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:06.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:06 smithi067 ceph-mon[33346]: pgmap v123: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 655 op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:06 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 31093/297996 objects degraded (10.434%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:06.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:07.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:07.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:06 smithi055 ceph-mon[29406]: pgmap v123: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 655 op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:06 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 31093/297996 objects degraded (10.434%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:06 smithi055 ceph-mon[40587]: pgmap v123: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 655 op/s; 31093/297996 objects degraded (10.434%) 2024-03-26T21:32:07.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:06 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 31093/297996 objects degraded (10.434%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.113 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:07.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:07.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:07.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:07.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:07.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:07.528 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:07.529 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:07.531 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:07.637 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:07.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:07.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 74 16 105596 105580 89.1542 62 0.0211119 0.0112118 2024-03-26T21:32:07.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:07.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[29406]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T21:32:07.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:07.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:07.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:07.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:07 smithi055 ceph-mon[40587]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T21:32:07.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:07.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:07.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:07 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:07 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:07 smithi067 ceph-mon[33346]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T21:32:08.043 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.094 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:08.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:08.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:08.233 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:08.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.381 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:08.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:08.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:08.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:08.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:08.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 75 15 106638 106623 88.8346 65.1875 0.0115372 0.0112519 2024-03-26T21:32:08.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:08.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:08.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:08.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:08.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:08.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:08 smithi055 ceph-mon[40587]: pgmap v124: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:08 smithi067 ceph-mon[33346]: pgmap v124: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:08 smithi055 ceph-mon[29406]: pgmap v124: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:08.991 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:09.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:09.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:09.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:09.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:09.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:09.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:09.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:09.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.619 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:09.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:09.714 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 76 16 107562 107546 88.4247 57.6875 0.0226343 0.0113043 2024-03-26T21:32:09.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:09.741 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:32:09.764 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:09.866 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:09.901 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:09.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:09 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:32:10.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:10.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:09 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:32:10.019 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:09 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:32:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:10.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.173 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:10.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:10.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:10.312 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:10.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:10.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:10.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:10.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:10.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:10.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 77 15 108461 108446 88.0069 56.25 0.0147485 0.0113578 2024-03-26T21:32:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:10.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:10.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:10.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:10 smithi067 ceph-mon[33346]: pgmap v126: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:10 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:32:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:10 smithi067 ceph-mon[33346]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T21:32:11.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:11.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[40587]: pgmap v126: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[40587]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[29406]: pgmap v126: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 32728/313359 objects degraded (10.444%) 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4021751094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:32:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:10 smithi055 ceph-mon[29406]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T21:32:11.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:11.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:11.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:11.286 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:11.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:11.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:11.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:11.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:11.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.680 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:11.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:11.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 78 16 109352 109336 87.5917 55.625 0.0136831 0.0114114 2024-03-26T21:32:11.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:11.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:11.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:11.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:12.020 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:32:12.020 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:32:12.020 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 32728/313359 objects degraded (10.444%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:12.020 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:32:12.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:32:12.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 32728/313359 objects degraded (10.444%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:12.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:12.066 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:12.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:12.194 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:11 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:32:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:11 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:32:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 32728/313359 objects degraded (10.444%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:12.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.338 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:12.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:12.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:12.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:12.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:12.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:12.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 79 16 110366 110350 87.2851 63.375 0.0139356 0.0114503 2024-03-26T21:32:12.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:12.760 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:12.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:12.888 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:12.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:12.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:13.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[29406]: pgmap v128: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 32728/318129 objects degraded (10.288%) 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[40587]: pgmap v128: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 32728/318129 objects degraded (10.288%) 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:32:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:12 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:13.122 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.168 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:12 smithi067 ceph-mon[33346]: pgmap v128: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 32728/318129 objects degraded (10.288%) 2024-03-26T21:32:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:12 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:32:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:12 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:32:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:12 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:32:13.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:13.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:13.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:13.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:13.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:13.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:13.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:13.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:32:13.720633+0000 min lat: 0.00119611 max lat: 0.190367 avg lat: 0.0115075 2024-03-26T21:32:13.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:32:13.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 80 16 111153 111137 86.8089 49.1875 0.00368228 0.0115075 2024-03-26T21:32:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:13.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:13.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:13 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:32:13.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:13 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:32:13.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:13 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:32:13.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:13 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:32:13.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:13.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:13.997 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:13.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:14.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:14.124 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:13 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:32:14.124 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:13 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:32:14.137 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:14.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:14.278 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:14.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:14.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:14.655 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:14.699 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:14.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 81 16 111955 111939 86.356 50.125 0.0249622 0.0115747 2024-03-26T21:32:14.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:14.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:14.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:14.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:14.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: pgmap v129: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 7.e scrub starts 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:32:15.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: pgmap v129: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 7.e scrub starts 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:32:15.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:14 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:32:15.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:15.138 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: pgmap v129: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:32:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 7.e scrub starts 2024-03-26T21:32:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:32:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:14 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:32:15.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:15.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:15.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:15.413 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:15.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:15.542 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:15.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:15.695 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:15.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 82 16 112878 112862 86.0064 57.6875 0.0107069 0.0116218 2024-03-26T21:32:15.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:15.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:15.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[40587]: 7.7 scrub starts 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[29406]: 7.7 scrub starts 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:32:16.056 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:15 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:16.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:16.104 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:16.206 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:15 smithi067 ceph-mon[33346]: 7.7 scrub starts 2024-03-26T21:32:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:15 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:32:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:15 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:32:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:15 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.248 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:16.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:16.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:16.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:16.556 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:16.658 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.703 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:16.712 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 83 16 113791 113775 85.6576 57.0625 0.00636856 0.0116695 2024-03-26T21:32:16.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:16.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:16.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:16.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:16.961 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:17.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:17.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: pgmap v130: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 607 op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 34232/327927 objects degraded (10.439%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: pgmap v130: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 607 op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:32:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:32:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:16 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 34232/327927 objects degraded (10.439%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:17.104 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:17.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:17.147 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:17.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:17.191 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:17.238 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:32:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:32:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: pgmap v130: 148 pgs: 27 active+undersized+degraded, 36 active+undersized, 85 active+clean; 6.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 607 op/s; 34232/327927 objects degraded (10.439%) 2024-03-26T21:32:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:32:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:32:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:16 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 34232/327927 objects degraded (10.439%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:17.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:17.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:17.338 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:17.383 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:17.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:17.426 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:17.427 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:17.449 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:17.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:17.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:17.713 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 84 16 114749 114733 85.3504 59.875 0.0161569 0.0117119 2024-03-26T21:32:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:17.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:17.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:17.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: 7.e scrub ok 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: 7.5 scrub starts 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: 7.7 scrub ok 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4212103422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: 7.e scrub ok 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:17.961 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:32:17.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:32:17.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: 7.5 scrub starts 2024-03-26T21:32:17.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: 7.7 scrub ok 2024-03-26T21:32:17.962 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:17 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4212103422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:17.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:18.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:18.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:18.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: 7.e scrub ok 2024-03-26T21:32:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:32:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:32:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: 7.5 scrub starts 2024-03-26T21:32:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: 7.7 scrub ok 2024-03-26T21:32:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:17 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4212103422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:18.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:18.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:18.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:32:18.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:18.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:18.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:18.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:32:18.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:32:18.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:18.714 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 85 16 115658 115642 85.0137 56.8125 0.0138868 0.0117573 2024-03-26T21:32:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.722 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:18.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:18.822 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:18.873 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[40587]: pgmap v131: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 952 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[40587]: 7.b scrub starts 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:32:18.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:32:18.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[29406]: pgmap v131: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 952 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:18.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-mon[29406]: 7.b scrub starts 2024-03-26T21:32:18.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:18 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:32:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:18 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:32:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:18 smithi067 ceph-mon[33346]: pgmap v131: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 952 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:18 smithi067 ceph-mon[33346]: 7.b scrub starts 2024-03-26T21:32:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:19.044 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:32:19.044 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:32:19.044 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:32:19.044 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:32:19.045 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:32:19.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:19.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:19.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:19.286 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:19.388 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:19.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:19.542 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:19.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:19.702 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:19.717 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 86 16 116375 116359 84.5462 44.8125 0.0154353 0.0118212 2024-03-26T21:32:19.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:19.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:32:19.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:19.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:19.872 INFO:teuthology.orchestra.run.smithi055.stderr:set 5.1a pg_upmap mapping to [5,6,0] 2024-03-26T21:32:19.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:20.005 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 7.5 scrub ok 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:32:20.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: 7.6 scrub starts 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]: dispatch 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 7.5 scrub ok 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:32:20.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:32:20.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T21:32:20.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: 7.6 scrub starts 2024-03-26T21:32:20.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]: dispatch 2024-03-26T21:32:20.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.054 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:19 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:20.136 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:20.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 7.5 scrub ok 2024-03-26T21:32:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: 7.6 scrub starts 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]: dispatch 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:19 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:20.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:20.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:20.433 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:20.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:20.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:32:20.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:20.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 87 16 117121 117105 84.1098 46.625 0.036849 0.0118837 2024-03-26T21:32:20.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.818 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:20.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:20.919 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:20.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:20.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:20.998 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: 7.a scrub starts 2024-03-26T21:32:20.998 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: pgmap v133: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 971 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]': finished 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: osdmap e121: 8 total, 7 up, 8 in 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: 7.f scrub starts 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[29406]: 7.b scrub ok 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: 7.a scrub starts 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: pgmap v133: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 971 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:32:20.999 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]': finished 2024-03-26T21:32:21.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: osdmap e121: 8 total, 7 up, 8 in 2024-03-26T21:32:21.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: 7.f scrub starts 2024-03-26T21:32:21.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:20 smithi055 ceph-mon[40587]: 7.b scrub ok 2024-03-26T21:32:21.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:21.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: 7.a scrub starts 2024-03-26T21:32:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: pgmap v133: 148 pgs: 2 active+clean+scrubbing, 27 active+undersized+degraded, 36 active+undersized, 83 active+clean; 6.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 971 op/s; 35703/341277 objects degraded (10.462%) 2024-03-26T21:32:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:32:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:32:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2871568438' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [5, 6, 0]}]': finished 2024-03-26T21:32:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: osdmap e121: 8 total, 7 up, 8 in 2024-03-26T21:32:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: 7.f scrub starts 2024-03-26T21:32:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:20 smithi067 ceph-mon[33346]: 7.b scrub ok 2024-03-26T21:32:21.257 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:21.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:21.388 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.429 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:21.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:21.457 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:21.523 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:21.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:21.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:21.713 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:21.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 88 16 117863 117847 83.681 46.375 0.0251353 0.0119452 2024-03-26T21:32:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:21.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:21.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:21.923 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:21.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:22.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.048 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:22.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:22.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:22.187 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:22.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:21 smithi067 ceph-mon[33346]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T21:32:22.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:21 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 35703/341277 objects degraded (10.462%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:22.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:21 smithi067 ceph-mon[33346]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T21:32:22.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:22.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[29406]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T21:32:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 35703/341277 objects degraded (10.462%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[29406]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T21:32:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[40587]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T21:32:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 35703/341277 objects degraded (10.462%), 27 pgs degraded, 63 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:21 smithi055 ceph-mon[40587]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T21:32:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.332 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:22.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:22.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.461 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:22.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:22.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:22.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:22.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:22.718 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 89 16 118576 118560 83.2414 44.5625 0.0211779 0.0120078 2024-03-26T21:32:22.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:22.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:22.880 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:22.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:22.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[29406]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 60 MiB/s wr, 956 op/s; 35703/345120 objects degraded (10.345%) 2024-03-26T21:32:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:32:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:32:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[29406]: 7.6 scrub ok 2024-03-26T21:32:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[29406]: 7.a scrub ok 2024-03-26T21:32:23.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[40587]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 60 MiB/s wr, 956 op/s; 35703/345120 objects degraded (10.345%) 2024-03-26T21:32:23.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:32:23.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:32:23.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[40587]: 7.6 scrub ok 2024-03-26T21:32:23.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:22 smithi055 ceph-mon[40587]: 7.a scrub ok 2024-03-26T21:32:23.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:23.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:22 smithi067 ceph-mon[33346]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 60 MiB/s wr, 956 op/s; 35703/345120 objects degraded (10.345%) 2024-03-26T21:32:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:22 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:32:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:22 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:32:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:22 smithi067 ceph-mon[33346]: 7.6 scrub ok 2024-03-26T21:32:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:22 smithi067 ceph-mon[33346]: 7.a scrub ok 2024-03-26T21:32:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:23.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:23.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:23.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:23.540 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:23.642 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:23.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total time run: 90.0126 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total writes made: 119484 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Write size: 65536 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object size: 65536 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Bandwidth (MB/sec): 82.9634 2024-03-26T21:32:23.715 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Bandwidth: 32.1076 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max bandwidth (MB/sec): 203 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min bandwidth (MB/sec): 44.5625 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average IOPS: 1327 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev IOPS: 513.722 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max IOPS: 3248 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min IOPS: 713 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average Latency(s): 0.0120491 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Latency(s): 0.0101888 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max latency(s): 0.190367 2024-03-26T21:32:23.716 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min latency(s): 0.00119611 2024-03-26T21:32:23.725 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T21:32:23.789 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:23.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:23.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:23.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:23.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:24.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:24.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:24.198 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:32:24.198 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:32:24.198 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:32:24.198 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:32:24.198 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[40587]: 7.f scrub ok 2024-03-26T21:32:24.199 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:32:24.199 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:32:24.199 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:32:24.199 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:32:24.199 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:23 smithi055 ceph-mon[29406]: 7.f scrub ok 2024-03-26T21:32:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:23 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:32:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:23 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:32:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:23 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:32:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:23 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:32:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:23 smithi067 ceph-mon[33346]: 7.f scrub ok 2024-03-26T21:32:24.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.277 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:24.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:24.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:24.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:24.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:24.677 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:24.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:24.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:24.949 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:24.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:25.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 910 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 910 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:32:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:32:25.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:25.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:24 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:25.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:25.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.221 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:25.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 910 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:24 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:25.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:25.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:25.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:25.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:25.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:25.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:25.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:25.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:25.879 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:25.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:25.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:26.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:26.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:26.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:26.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:25 smithi055 ceph-mon[29406]: 7.d scrub starts 2024-03-26T21:32:26.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:25 smithi055 ceph-mon[29406]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T21:32:26.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:25 smithi055 ceph-mon[40587]: 7.d scrub starts 2024-03-26T21:32:26.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:25 smithi055 ceph-mon[40587]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T21:32:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:25 smithi067 ceph-mon[33346]: 7.d scrub starts 2024-03-26T21:32:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:25 smithi067 ceph-mon[33346]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T21:32:26.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:26.273 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:26.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:26.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T21:32:26.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:32:26.380 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T21:32:26.380 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T21:32:26.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:32:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.429 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:26.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:26.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.595 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:26.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:26.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:26.749 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:26.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:26.889 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:26.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:26.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:26.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:27.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:27.176 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-mon[33346]: pgmap v140: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 936 B/s rd, 47 MiB/s wr, 749 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:32:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:32:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 37002/353562 objects degraded (10.465%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:26 smithi067 ceph-mon[33346]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T21:32:27.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:27.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[40587]: pgmap v140: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 936 B/s rd, 47 MiB/s wr, 749 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:27.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:32:27.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 37002/353562 objects degraded (10.465%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[40587]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[29406]: pgmap v140: 148 pgs: 3 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 83 active+clean; 7.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 936 B/s rd, 47 MiB/s wr, 749 op/s; 37002/353562 objects degraded (10.465%) 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 37002/353562 objects degraded (10.465%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:27.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:26 smithi055 ceph-mon[29406]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T21:32:27.321 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:27.422 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:27.463 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:27.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:27.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:27.742 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:27.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:27.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:27.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:27.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:28.051 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 7.d scrub ok 2024-03-26T21:32:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 7.c scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:27 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 7.d scrub ok 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 7.c scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:32:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:32:28.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 7.d scrub ok 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 7.c scrub starts 2024-03-26T21:32:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:32:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:27 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:32:28.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:28.330 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:28.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.489 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:28.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:28.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:28.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:28.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:28.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:28.785 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:28.824 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:28.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:28.876 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:28.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:28.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:28.926 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:28.968 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:28.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:29.008 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:29.051 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:29.051 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:29.051 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:29.052 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:29.124 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:29.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:29.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:29.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[40587]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 1.25k op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[40587]: 7.c scrub ok 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1015207163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[29406]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 1.25k op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[29406]: 7.c scrub ok 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1015207163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:29.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:29.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:29.426 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:29 smithi067 ceph-mon[33346]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 1.25k op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:29 smithi067 ceph-mon[33346]: 7.c scrub ok 2024-03-26T21:32:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1015207163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:29.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:29.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:29.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:29.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:29.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:29.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:29.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:29.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:30.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:30.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:30.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:30.268 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:30.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:32:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:30 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:32:30.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:30.409 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:30 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:32:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:30 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:32:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:30 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:32:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:30 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:32:30.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:30.547 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:30.649 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:30.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:30.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:30.822 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:30.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:30.963 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:30.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:31.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:31.092 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:31.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:31.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:31.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 788 B/s rd, 23 MiB/s wr, 739 op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 788 B/s rd, 23 MiB/s wr, 739 op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:32:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:31 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:31.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:31.377 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:31.479 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 7.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 788 B/s rd, 23 MiB/s wr, 739 op/s; 36668/350424 objects degraded (10.464%) 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:32:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:31 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:31.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:31.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:31.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:31.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:31.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:31.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:31.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:32.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:32.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:32.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:32.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:32.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:32.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:32:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:32:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 36668/350424 objects degraded (10.464%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:32:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:32:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 36668/350424 objects degraded (10.464%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:32.334 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:32.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:32.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:32 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:32:32.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:32 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:32:32.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 36668/350424 objects degraded (10.464%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:32.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.595 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:32.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:32.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.739 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:32.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:32.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:32.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:32.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:32.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:33.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:33.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:33.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:33.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:33.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:33.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:33 smithi055 ceph-mon[40587]: pgmap v144: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 7.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 14 MiB/s wr, 794 op/s; 36668/345483 objects degraded (10.614%) 2024-03-26T21:32:33.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:33 smithi055 ceph-mon[29406]: pgmap v144: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 7.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 14 MiB/s wr, 794 op/s; 36668/345483 objects degraded (10.614%) 2024-03-26T21:32:33.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:33.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:33 smithi067 ceph-mon[33346]: pgmap v144: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 7.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 14 MiB/s wr, 794 op/s; 36668/345483 objects degraded (10.614%) 2024-03-26T21:32:33.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:33.543 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:33.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:33.672 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:33.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:33.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:33.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:33.922 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:32:33.922 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:32:33.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:33.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:33.977 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:34.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:32:34.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:32:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.099 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:34.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:34.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:34.261 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:34.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:34 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:32:34.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:34 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:32:34.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:34 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:32:34.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:34 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:32:34.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.410 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:34.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:34 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:32:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:34 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:32:34.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:34.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:34.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:34.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:34.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:34.832 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:34.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:34.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:34.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:35.062 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:35.128 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:32:35.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:35.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:35.324 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:35.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:35.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[40587]: pgmap v145: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 14 MiB/s wr, 1.20k op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:35.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/595307796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[40587]: 7.2 scrub starts 2024-03-26T21:32:35.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[29406]: pgmap v145: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 14 MiB/s wr, 1.20k op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:35.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/595307796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.381 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.381 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:35 smithi055 ceph-mon[29406]: 7.2 scrub starts 2024-03-26T21:32:35.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:32:35.450 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:35 smithi067 ceph-mon[33346]: pgmap v145: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 14 MiB/s wr, 1.20k op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:35.450 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/595307796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.450 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:35 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:35.450 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:35 smithi067 ceph-mon[33346]: 7.2 scrub starts 2024-03-26T21:32:35.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:35.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:35.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:35.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:35.732 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:35.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:35.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:35.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:35.919 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:35.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:36.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:36.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:32:36.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[29406]: 7.0 scrub starts 2024-03-26T21:32:36.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:32:36.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:36.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[29406]: osdmap e126: 8 total, 7 up, 8 in 2024-03-26T21:32:36.111 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[40587]: 7.0 scrub starts 2024-03-26T21:32:36.112 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:32:36.112 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:36.112 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:36 smithi055 ceph-mon[40587]: osdmap e126: 8 total, 7 up, 8 in 2024-03-26T21:32:36.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:36.166 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:36 smithi067 ceph-mon[33346]: 7.0 scrub starts 2024-03-26T21:32:36.167 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:36 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:32:36.167 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:36 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:36.167 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:36 smithi067 ceph-mon[33346]: osdmap e126: 8 total, 7 up, 8 in 2024-03-26T21:32:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.184 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:36.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:36.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:36.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:36.432 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:36.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:36.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.630 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:36.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:36.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.783 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:36.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:36.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:36.918 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:36.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:36.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:37.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:37.108 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: pgmap v147: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 607 op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.140 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:37.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: pgmap v147: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 607 op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: 3.5 deep-scrub starts 2024-03-26T21:32:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: 3.5 deep-scrub ok 2024-03-26T21:32:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: 7.0 scrub ok 2024-03-26T21:32:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:32:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:32:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 34958/334428 objects degraded (10.453%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:37 smithi067 ceph-mon[33346]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T21:32:37.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:37.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: 3.5 deep-scrub starts 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: 3.5 deep-scrub ok 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: 7.0 scrub ok 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:32:37.396 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 34958/334428 objects degraded (10.453%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[29406]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: pgmap v147: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 607 op/s; 34958/334428 objects degraded (10.453%) 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: 3.5 deep-scrub starts 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: 3.5 deep-scrub ok 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: 7.0 scrub ok 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 34958/334428 objects degraded (10.453%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:37.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:37 smithi055 ceph-mon[40587]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T21:32:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:37.490 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:37.503 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:37.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:37.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:37.685 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:32:37.685 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:32:37.732 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:37.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:37.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:37.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:37.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:37.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:38.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:38.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:38.207 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: 7.2 scrub ok 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:32:38.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:32:38.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: 7.2 scrub ok 2024-03-26T21:32:38.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:38.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:38.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:38.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:38.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:38.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:38.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: 7.2 scrub ok 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:32:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:32:38.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:38 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:32:38.539 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:38.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:38.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:32:38.678 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:38.718 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:38.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:38.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:38.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:32:38.929 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:38.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:39.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:39.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:39 smithi067 ceph-mon[33346]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:39 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:32:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:39 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:32:39.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:39.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:39.286 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[29406]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:39.286 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:32:39.286 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:32:39.287 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:39.287 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:32:39.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[40587]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:39.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:32:39.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:39 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:32:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:32:39.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:39.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:39.556 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:39.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:39.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:39.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:39.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:39.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:39.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:39.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.009 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:40.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:40.066 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-03-26T21:32:37.350359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6970677696,"num_objects":106554,"num_object_clones":0,"num_object_copies":319662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106554,"num_whiteouts":0,"num_read":1162,"num_read_kb":1008,"num_write":133229,"num_write_kb":7649438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89977,"ondisk_log_size":89977,"up":382,"acting":382,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":440,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19097544,"kb_used_data":18392560,"kb_used_omap":12,"kb_used_meta":704691,"kb_avail":730863672,"statfs":{"total":767960285184,"available":748404400128,"internally_reserved":0,"allocated":18833981440,"data_stored":18829994951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":721604144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-672006144,"num_objects":-10254,"num_object_clones":0,"num_object_copies":-30762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10254,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10254,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004759"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.117455+0000","last_change":"2024-03-26T21:30:31.738275+0000","last_active":"2024-03-26T21:32:35.117455+0000","last_peered":"2024-03-26T21:32:35.117455+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738275+0000","last_became_peered":"2024-03-26T21:30:31.738275+0000","last_unstale":"2024-03-26T21:32:35.117455+0000","last_undegraded":"2024-03-26T21:32:35.117455+0000","last_fullsized":"2024-03-26T21:30:31.732719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116710+0000","last_change":"2024-03-26T21:32:14.025645+0000","last_active":"2024-03-26T21:32:35.116710+0000","last_peered":"2024-03-26T21:32:35.116710+0000","last_clean":"2024-03-26T21:32:35.116710+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:32:35.116710+0000","last_undegraded":"2024-03-26T21:32:35.116710+0000","last_fullsized":"2024-03-26T21:32:35.116710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:14.025432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:14.025432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:31.913882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120622+0000","last_change":"2024-03-26T21:32:14.926305+0000","last_active":"2024-03-26T21:32:35.120622+0000","last_peered":"2024-03-26T21:32:35.120622+0000","last_clean":"2024-03-26T21:32:35.120622+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:32:35.120622+0000","last_undegraded":"2024-03-26T21:32:35.120622+0000","last_fullsized":"2024-03-26T21:32:35.120622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:14.926185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:32:14.926185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:18.655858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":108,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.105589+0000","last_change":"2024-03-26T21:30:31.740069+0000","last_active":"2024-03-26T21:32:35.105589+0000","last_peered":"2024-03-26T21:32:35.105589+0000","last_clean":"2024-03-26T21:28:42.030750+0000","last_became_active":"2024-03-26T21:30:31.740069+0000","last_became_peered":"2024-03-26T21:30:31.740069+0000","last_unstale":"2024-03-26T21:32:35.105589+0000","last_undegraded":"2024-03-26T21:30:31.731923+0000","last_fullsized":"2024-03-26T21:30:31.731769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:51.544608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:51.544608+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116928+0000","last_change":"2024-03-26T21:31:16.924155+0000","last_active":"2024-03-26T21:32:35.116928+0000","last_peered":"2024-03-26T21:32:35.116928+0000","last_clean":"2024-03-26T21:32:35.116928+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:32:35.116928+0000","last_undegraded":"2024-03-26T21:32:35.116928+0000","last_fullsized":"2024-03-26T21:32:35.116928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:16.923867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:16.923867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.105325+0000","last_change":"2024-03-26T21:30:31.739659+0000","last_active":"2024-03-26T21:32:35.105325+0000","last_peered":"2024-03-26T21:32:35.105325+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.739659+0000","last_became_peered":"2024-03-26T21:30:31.739659+0000","last_unstale":"2024-03-26T21:32:35.105325+0000","last_undegraded":"2024-03-26T21:32:35.105325+0000","last_fullsized":"2024-03-26T21:30:31.731768+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.645565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.645565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":147,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117921+0000","last_change":"2024-03-26T21:31:17.917878+0000","last_active":"2024-03-26T21:32:35.117921+0000","last_peered":"2024-03-26T21:32:35.117921+0000","last_clean":"2024-03-26T21:32:35.117921+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:32:35.117921+0000","last_undegraded":"2024-03-26T21:32:35.117921+0000","last_fullsized":"2024-03-26T21:32:35.117921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:17.917762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:17.917762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.999530+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":167,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119559+0000","last_change":"2024-03-26T21:32:13.108197+0000","last_active":"2024-03-26T21:32:35.119559+0000","last_peered":"2024-03-26T21:32:35.119559+0000","last_clean":"2024-03-26T21:32:35.119559+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:32:35.119559+0000","last_undegraded":"2024-03-26T21:32:35.119559+0000","last_fullsized":"2024-03-26T21:32:35.119559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:13.107771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.107771+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:24.857568+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117186+0000","last_change":"2024-03-26T21:31:42.540278+0000","last_active":"2024-03-26T21:32:35.117186+0000","last_peered":"2024-03-26T21:32:35.117186+0000","last_clean":"2024-03-26T21:32:35.117186+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:32:35.117186+0000","last_undegraded":"2024-03-26T21:32:35.117186+0000","last_fullsized":"2024-03-26T21:32:35.117186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:42.540161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:42.540161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:01.295047+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117595+0000","last_change":"2024-03-26T21:32:26.594505+0000","last_active":"2024-03-26T21:32:35.117595+0000","last_peered":"2024-03-26T21:32:35.117595+0000","last_clean":"2024-03-26T21:32:35.117595+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:32:35.117595+0000","last_undegraded":"2024-03-26T21:32:35.117595+0000","last_fullsized":"2024-03-26T21:32:35.117595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.594348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.594348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:30.758140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118206+0000","last_change":"2024-03-26T21:30:31.738978+0000","last_active":"2024-03-26T21:32:35.118206+0000","last_peered":"2024-03-26T21:32:35.118206+0000","last_clean":"2024-03-26T21:30:22.526419+0000","last_became_active":"2024-03-26T21:30:31.738978+0000","last_became_peered":"2024-03-26T21:30:31.738978+0000","last_unstale":"2024-03-26T21:32:35.118206+0000","last_undegraded":"2024-03-26T21:32:35.118206+0000","last_fullsized":"2024-03-26T21:30:31.733643+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:22.526329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:22.526329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":108,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.143138+0000","last_change":"2024-03-26T21:30:31.742645+0000","last_active":"2024-03-26T21:32:35.143138+0000","last_peered":"2024-03-26T21:32:35.143138+0000","last_clean":"2024-03-26T21:28:42.145195+0000","last_became_active":"2024-03-26T21:30:31.742645+0000","last_became_peered":"2024-03-26T21:30:31.742645+0000","last_unstale":"2024-03-26T21:32:35.143138+0000","last_undegraded":"2024-03-26T21:30:31.730765+0000","last_fullsized":"2024-03-26T21:30:31.730641+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:29:52.496706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:52.496706+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117194+0000","last_change":"2024-03-26T21:32:13.527807+0000","last_active":"2024-03-26T21:32:35.117194+0000","last_peered":"2024-03-26T21:32:35.117194+0000","last_clean":"2024-03-26T21:32:35.117194+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:32:35.117194+0000","last_undegraded":"2024-03-26T21:32:35.117194+0000","last_fullsized":"2024-03-26T21:32:35.117194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:13.527732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.527732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:20.658244+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":27,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117222+0000","last_change":"2024-03-26T21:32:23.533404+0000","last_active":"2024-03-26T21:32:35.117222+0000","last_peered":"2024-03-26T21:32:35.117222+0000","last_clean":"2024-03-26T21:32:35.117222+0000","last_became_active":"2024-03-26T21:32:20.904068+0000","last_became_peered":"2024-03-26T21:32:20.904068+0000","last_unstale":"2024-03-26T21:32:35.117222+0000","last_undegraded":"2024-03-26T21:32:35.117222+0000","last_fullsized":"2024-03-26T21:32:35.117222+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.533287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.533287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:24.359654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.117880+0000","last_change":"2024-03-26T21:30:31.738170+0000","last_active":"2024-03-26T21:32:35.117880+0000","last_peered":"2024-03-26T21:32:35.117880+0000","last_clean":"2024-03-26T21:30:28.546608+0000","last_became_active":"2024-03-26T21:30:31.738170+0000","last_became_peered":"2024-03-26T21:30:31.738170+0000","last_unstale":"2024-03-26T21:32:35.117880+0000","last_undegraded":"2024-03-26T21:32:35.117880+0000","last_fullsized":"2024-03-26T21:30:31.731908+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:28.546501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:28.546501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":102,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.116421+0000","last_change":"2024-03-26T21:30:31.745243+0000","last_active":"2024-03-26T21:32:35.116421+0000","last_peered":"2024-03-26T21:32:35.116421+0000","last_clean":"2024-03-26T21:28:42.123479+0000","last_became_active":"2024-03-26T21:30:31.745243+0000","last_became_peered":"2024-03-26T21:30:31.745243+0000","last_unstale":"2024-03-26T21:32:35.116421+0000","last_undegraded":"2024-03-26T21:30:31.731435+0000","last_fullsized":"2024-03-26T21:30:31.731307+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.105706+0000","last_change":"2024-03-26T21:30:31.740216+0000","last_active":"2024-03-26T21:32:35.105706+0000","last_peered":"2024-03-26T21:32:35.105706+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.740216+0000","last_became_peered":"2024-03-26T21:30:31.740216+0000","last_unstale":"2024-03-26T21:32:35.105706+0000","last_undegraded":"2024-03-26T21:32:35.105706+0000","last_fullsized":"2024-03-26T21:30:31.732999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:28:39.845082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.120557+0000","last_change":"2024-03-26T21:30:31.748769+0000","last_active":"2024-03-26T21:32:35.120557+0000","last_peered":"2024-03-26T21:32:35.120557+0000","last_clean":"2024-03-26T21:30:19.057549+0000","last_became_active":"2024-03-26T21:30:31.748769+0000","last_became_peered":"2024-03-26T21:30:31.748769+0000","last_unstale":"2024-03-26T21:32:35.120557+0000","last_undegraded":"2024-03-26T21:32:35.120557+0000","last_fullsized":"2024-03-26T21:30:31.734303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.051556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.051556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118414+0000","last_change":"2024-03-26T21:30:31.744225+0000","last_active":"2024-03-26T21:32:35.118414+0000","last_peered":"2024-03-26T21:32:35.118414+0000","last_clean":"2024-03-26T21:30:19.056406+0000","last_became_active":"2024-03-26T21:30:31.744225+0000","last_became_peered":"2024-03-26T21:30:31.744225+0000","last_unstale":"2024-03-26T21:32:35.118414+0000","last_undegraded":"2024-03-26T21:32:35.118414+0000","last_fullsized":"2024-03-26T21:30:31.737824+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.708767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.708767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":175,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.129029+0000","last_change":"2024-03-26T21:32:29.620716+0000","last_active":"2024-03-26T21:32:35.129029+0000","last_peered":"2024-03-26T21:32:35.129029+0000","last_clean":"2024-03-26T21:32:35.129029+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:32:35.129029+0000","last_undegraded":"2024-03-26T21:32:35.129029+0000","last_fullsized":"2024-03-26T21:32:35.129029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:29.620574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.620574+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:50.318752+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120018+0000","last_change":"2024-03-26T21:32:30.360160+0000","last_active":"2024-03-26T21:32:35.120018+0000","last_peered":"2024-03-26T21:32:35.120018+0000","last_clean":"2024-03-26T21:32:35.120018+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:32:35.120018+0000","last_undegraded":"2024-03-26T21:32:35.120018+0000","last_fullsized":"2024-03-26T21:32:35.120018+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.360050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.360050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:33.757790+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.119409+0000","last_change":"2024-03-26T21:30:31.741526+0000","last_active":"2024-03-26T21:32:35.119409+0000","last_peered":"2024-03-26T21:32:35.119409+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.741526+0000","last_became_peered":"2024-03-26T21:30:31.741526+0000","last_unstale":"2024-03-26T21:32:35.119409+0000","last_undegraded":"2024-03-26T21:32:35.119409+0000","last_fullsized":"2024-03-26T21:30:31.731261+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:44.615910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.615910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.122262+0000","last_change":"2024-03-26T21:30:31.740626+0000","last_active":"2024-03-26T21:32:35.122262+0000","last_peered":"2024-03-26T21:32:35.122262+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740626+0000","last_became_peered":"2024-03-26T21:30:31.740626+0000","last_unstale":"2024-03-26T21:32:35.122262+0000","last_undegraded":"2024-03-26T21:32:35.122262+0000","last_fullsized":"2024-03-26T21:30:31.732923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":152,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.116627+0000","last_change":"2024-03-26T21:30:31.745104+0000","last_active":"2024-03-26T21:32:35.116627+0000","last_peered":"2024-03-26T21:32:35.116627+0000","last_clean":"2024-03-26T21:30:19.791949+0000","last_became_active":"2024-03-26T21:30:31.745104+0000","last_became_peered":"2024-03-26T21:30:31.745104+0000","last_unstale":"2024-03-26T21:32:35.116627+0000","last_undegraded":"2024-03-26T21:30:31.729821+0000","last_fullsized":"2024-03-26T21:30:31.729686+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128444+0000","last_change":"2024-03-26T21:31:14.781495+0000","last_active":"2024-03-26T21:32:35.128444+0000","last_peered":"2024-03-26T21:32:35.128444+0000","last_clean":"2024-03-26T21:32:35.128444+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:32:35.128444+0000","last_undegraded":"2024-03-26T21:32:35.128444+0000","last_fullsized":"2024-03-26T21:32:35.128444+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:14.781351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.781351+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.404865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":194,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.106134+0000","last_change":"2024-03-26T21:30:31.738596+0000","last_active":"2024-03-26T21:32:35.106134+0000","last_peered":"2024-03-26T21:32:35.106134+0000","last_clean":"2024-03-26T21:30:29.906831+0000","last_became_active":"2024-03-26T21:30:31.738596+0000","last_became_peered":"2024-03-26T21:30:31.738596+0000","last_unstale":"2024-03-26T21:32:35.106134+0000","last_undegraded":"2024-03-26T21:30:31.733688+0000","last_fullsized":"2024-03-26T21:30:31.733580+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:10.722203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:10.722203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":143,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.116808+0000","last_change":"2024-03-26T21:30:31.740769+0000","last_active":"2024-03-26T21:32:35.116808+0000","last_peered":"2024-03-26T21:32:35.116808+0000","last_clean":"2024-03-26T21:30:19.055952+0000","last_became_active":"2024-03-26T21:30:31.740769+0000","last_became_peered":"2024-03-26T21:30:31.740769+0000","last_unstale":"2024-03-26T21:32:35.116808+0000","last_undegraded":"2024-03-26T21:30:31.730676+0000","last_fullsized":"2024-03-26T21:30:31.730563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.058714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.122187+0000","last_change":"2024-03-26T21:32:22.112186+0000","last_active":"2024-03-26T21:32:35.122187+0000","last_peered":"2024-03-26T21:32:35.122187+0000","last_clean":"2024-03-26T21:32:35.122187+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:32:35.122187+0000","last_undegraded":"2024-03-26T21:32:35.122187+0000","last_fullsized":"2024-03-26T21:32:35.122187+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:32:22.112001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.112001+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:42.509988+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.117408+0000","last_change":"2024-03-26T21:30:31.739460+0000","last_active":"2024-03-26T21:32:35.117408+0000","last_peered":"2024-03-26T21:32:35.117408+0000","last_clean":"2024-03-26T21:30:19.057225+0000","last_became_active":"2024-03-26T21:30:31.739460+0000","last_became_peered":"2024-03-26T21:30:31.739460+0000","last_unstale":"2024-03-26T21:32:35.117408+0000","last_undegraded":"2024-03-26T21:32:35.117408+0000","last_fullsized":"2024-03-26T21:30:31.733695+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:45.092616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:45.092616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.122752+0000","last_change":"2024-03-26T21:31:28.707417+0000","last_active":"2024-03-26T21:32:35.122752+0000","last_peered":"2024-03-26T21:32:35.122752+0000","last_clean":"2024-03-26T21:32:35.122752+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:32:35.122752+0000","last_undegraded":"2024-03-26T21:32:35.122752+0000","last_fullsized":"2024-03-26T21:32:35.122752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.707018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.375332+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119077+0000","last_change":"2024-03-26T21:31:39.883235+0000","last_active":"2024-03-26T21:32:35.119077+0000","last_peered":"2024-03-26T21:32:35.119077+0000","last_clean":"2024-03-26T21:32:35.119077+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:32:35.119077+0000","last_undegraded":"2024-03-26T21:32:35.119077+0000","last_fullsized":"2024-03-26T21:32:35.119077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.883105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.883105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:03.189316+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.106031+0000","last_change":"2024-03-26T21:31:23.609409+0000","last_active":"2024-03-26T21:32:35.106031+0000","last_peered":"2024-03-26T21:32:35.106031+0000","last_clean":"2024-03-26T21:32:35.106031+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:32:35.106031+0000","last_undegraded":"2024-03-26T21:32:35.106031+0000","last_fullsized":"2024-03-26T21:32:35.106031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:31:23.609259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:23.609259+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127681+0000","last_change":"2024-03-26T21:30:31.738804+0000","last_active":"2024-03-26T21:32:35.127681+0000","last_peered":"2024-03-26T21:32:35.127681+0000","last_clean":"2024-03-26T21:30:19.144401+0000","last_became_active":"2024-03-26T21:30:31.738804+0000","last_became_peered":"2024-03-26T21:30:31.738804+0000","last_unstale":"2024-03-26T21:32:35.127681+0000","last_undegraded":"2024-03-26T21:32:35.127681+0000","last_fullsized":"2024-03-26T21:30:31.735055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.535399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.535399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.106135+0000","last_change":"2024-03-26T21:30:31.739828+0000","last_active":"2024-03-26T21:32:35.106135+0000","last_peered":"2024-03-26T21:32:35.106135+0000","last_clean":"2024-03-26T21:30:19.905962+0000","last_became_active":"2024-03-26T21:30:31.739828+0000","last_became_peered":"2024-03-26T21:30:31.739828+0000","last_unstale":"2024-03-26T21:32:35.106135+0000","last_undegraded":"2024-03-26T21:32:35.106135+0000","last_fullsized":"2024-03-26T21:30:31.731999+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.749046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.749046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120830+0000","last_change":"2024-03-26T21:31:26.716901+0000","last_active":"2024-03-26T21:32:35.120830+0000","last_peered":"2024-03-26T21:32:35.120830+0000","last_clean":"2024-03-26T21:32:35.120830+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:32:35.120830+0000","last_undegraded":"2024-03-26T21:32:35.120830+0000","last_fullsized":"2024-03-26T21:32:35.120830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:26.716779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:26.716779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.514181+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120383+0000","last_change":"2024-03-26T21:31:25.930138+0000","last_active":"2024-03-26T21:32:35.120383+0000","last_peered":"2024-03-26T21:32:35.120383+0000","last_clean":"2024-03-26T21:32:35.120383+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:32:35.120383+0000","last_undegraded":"2024-03-26T21:32:35.120383+0000","last_fullsized":"2024-03-26T21:32:35.120383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:31:25.930008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.930008+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.183279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119440+0000","last_change":"2024-03-26T21:32:29.393768+0000","last_active":"2024-03-26T21:32:35.119440+0000","last_peered":"2024-03-26T21:32:35.119440+0000","last_clean":"2024-03-26T21:32:35.119440+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:32:35.119440+0000","last_undegraded":"2024-03-26T21:32:35.119440+0000","last_fullsized":"2024-03-26T21:32:35.119440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:29.393601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.393601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:58.178504+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127866+0000","last_change":"2024-03-26T21:30:31.742998+0000","last_active":"2024-03-26T21:32:35.127866+0000","last_peered":"2024-03-26T21:32:35.127866+0000","last_clean":"2024-03-26T21:30:19.144660+0000","last_became_active":"2024-03-26T21:30:31.742998+0000","last_became_peered":"2024-03-26T21:30:31.742998+0000","last_unstale":"2024-03-26T21:32:35.127866+0000","last_undegraded":"2024-03-26T21:32:35.127866+0000","last_fullsized":"2024-03-26T21:30:31.735078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118305+0000","last_change":"2024-03-26T21:31:37.579162+0000","last_active":"2024-03-26T21:32:35.118305+0000","last_peered":"2024-03-26T21:32:35.118305+0000","last_clean":"2024-03-26T21:32:35.118305+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:32:35.118305+0000","last_undegraded":"2024-03-26T21:32:35.118305+0000","last_fullsized":"2024-03-26T21:32:35.118305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.579051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.579051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:43.787885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119452+0000","last_change":"2024-03-26T21:32:15.138279+0000","last_active":"2024-03-26T21:32:35.119452+0000","last_peered":"2024-03-26T21:32:35.119452+0000","last_clean":"2024-03-26T21:32:35.119452+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:32:35.119452+0000","last_undegraded":"2024-03-26T21:32:35.119452+0000","last_fullsized":"2024-03-26T21:32:35.119452+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:32:15.138133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:15.138133+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:15.178921+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116874+0000","last_change":"2024-03-26T21:32:26.906415+0000","last_active":"2024-03-26T21:32:35.116874+0000","last_peered":"2024-03-26T21:32:35.116874+0000","last_clean":"2024-03-26T21:32:35.116874+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:32:35.116874+0000","last_undegraded":"2024-03-26T21:32:35.116874+0000","last_fullsized":"2024-03-26T21:32:35.116874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.906307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.906307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:39.098534+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116834+0000","last_change":"2024-03-26T21:31:32.871165+0000","last_active":"2024-03-26T21:32:35.116834+0000","last_peered":"2024-03-26T21:32:35.116834+0000","last_clean":"2024-03-26T21:32:35.116834+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:32:35.116834+0000","last_undegraded":"2024-03-26T21:32:35.116834+0000","last_fullsized":"2024-03-26T21:32:35.116834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:32.871039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.871039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:53.774307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.128103+0000","last_change":"2024-03-26T21:30:31.736143+0000","last_active":"2024-03-26T21:32:35.128103+0000","last_peered":"2024-03-26T21:32:35.128103+0000","last_clean":"2024-03-26T21:30:19.144685+0000","last_became_active":"2024-03-26T21:30:31.736143+0000","last_became_peered":"2024-03-26T21:30:31.736143+0000","last_unstale":"2024-03-26T21:32:35.128103+0000","last_undegraded":"2024-03-26T21:32:35.128103+0000","last_fullsized":"2024-03-26T21:30:31.733617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:55.577488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.577488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.122919+0000","last_change":"2024-03-26T21:31:36.082385+0000","last_active":"2024-03-26T21:32:35.122919+0000","last_peered":"2024-03-26T21:32:35.122919+0000","last_clean":"2024-03-26T21:32:35.122919+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:32:35.122919+0000","last_undegraded":"2024-03-26T21:32:35.122919+0000","last_fullsized":"2024-03-26T21:32:35.122919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:31:36.082236+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:31:36.082236+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.095627+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117799+0000","last_change":"2024-03-26T21:31:38.898103+0000","last_active":"2024-03-26T21:32:35.117799+0000","last_peered":"2024-03-26T21:32:35.117799+0000","last_clean":"2024-03-26T21:32:35.117799+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:32:35.117799+0000","last_undegraded":"2024-03-26T21:32:35.117799+0000","last_fullsized":"2024-03-26T21:32:35.117799+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:38.897810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.897810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:50.848439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.122565+0000","last_change":"2024-03-26T21:31:33.046592+0000","last_active":"2024-03-26T21:32:35.122565+0000","last_peered":"2024-03-26T21:32:35.122565+0000","last_clean":"2024-03-26T21:32:35.122565+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:32:35.122565+0000","last_undegraded":"2024-03-26T21:32:35.122565+0000","last_fullsized":"2024-03-26T21:32:35.122565+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:33.046470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:33.046470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.166286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.128063+0000","last_change":"2024-03-26T21:30:31.741210+0000","last_active":"2024-03-26T21:32:35.128063+0000","last_peered":"2024-03-26T21:32:35.128063+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.741210+0000","last_became_peered":"2024-03-26T21:30:31.741210+0000","last_unstale":"2024-03-26T21:32:35.128063+0000","last_undegraded":"2024-03-26T21:32:35.128063+0000","last_fullsized":"2024-03-26T21:30:31.734566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:26.155650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:26.155650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.790632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":111,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.106423+0000","last_change":"2024-03-26T21:30:31.741165+0000","last_active":"2024-03-26T21:32:35.106423+0000","last_peered":"2024-03-26T21:32:35.106423+0000","last_clean":"2024-03-26T21:28:42.207254+0000","last_became_active":"2024-03-26T21:30:31.741165+0000","last_became_peered":"2024-03-26T21:30:31.741165+0000","last_unstale":"2024-03-26T21:32:35.106423+0000","last_undegraded":"2024-03-26T21:30:31.733400+0000","last_fullsized":"2024-03-26T21:30:31.733163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120682+0000","last_change":"2024-03-26T21:31:40.605350+0000","last_active":"2024-03-26T21:32:35.120682+0000","last_peered":"2024-03-26T21:32:35.120682+0000","last_clean":"2024-03-26T21:32:35.120682+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:32:35.120682+0000","last_undegraded":"2024-03-26T21:32:35.120682+0000","last_fullsized":"2024-03-26T21:32:35.120682+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:40.605038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:40.605038+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.927678+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":196,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118223+0000","last_change":"2024-03-26T21:31:19.736062+0000","last_active":"2024-03-26T21:32:35.118223+0000","last_peered":"2024-03-26T21:32:35.118223+0000","last_clean":"2024-03-26T21:32:35.118223+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:32:35.118223+0000","last_undegraded":"2024-03-26T21:32:35.118223+0000","last_fullsized":"2024-03-26T21:32:35.118223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:31:19.735944+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:31:19.735944+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.497485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127994+0000","last_change":"2024-03-26T21:30:31.738173+0000","last_active":"2024-03-26T21:32:35.127994+0000","last_peered":"2024-03-26T21:32:35.127994+0000","last_clean":"2024-03-26T21:30:19.144537+0000","last_became_active":"2024-03-26T21:30:31.738173+0000","last_became_peered":"2024-03-26T21:30:31.738173+0000","last_unstale":"2024-03-26T21:32:35.127994+0000","last_undegraded":"2024-03-26T21:32:35.127994+0000","last_fullsized":"2024-03-26T21:30:31.733359+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.550817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.550817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119813+0000","last_change":"2024-03-26T21:31:28.999375+0000","last_active":"2024-03-26T21:32:35.119813+0000","last_peered":"2024-03-26T21:32:35.119813+0000","last_clean":"2024-03-26T21:32:35.119813+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:32:35.119813+0000","last_undegraded":"2024-03-26T21:32:35.119813+0000","last_fullsized":"2024-03-26T21:32:35.119813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:31:28.998501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.998501+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.965671+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.129438+0000","last_change":"2024-03-26T21:30:31.744131+0000","last_active":"2024-03-26T21:32:35.129438+0000","last_peered":"2024-03-26T21:32:35.129438+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.744131+0000","last_became_peered":"2024-03-26T21:30:31.744131+0000","last_unstale":"2024-03-26T21:32:35.129438+0000","last_undegraded":"2024-03-26T21:32:35.129438+0000","last_fullsized":"2024-03-26T21:30:31.735473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":198,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.313184+0000","last_change":"2024-03-26T21:32:36.313231+0000","last_active":"2024-03-26T21:32:36.313184+0000","last_peered":"2024-03-26T21:32:36.313184+0000","last_clean":"2024-03-26T21:32:36.313184+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:32:36.313184+0000","last_undegraded":"2024-03-26T21:32:36.313184+0000","last_fullsized":"2024-03-26T21:32:36.313184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:36.311788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.311788+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:53.867297+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117723+0000","last_change":"2024-03-26T21:32:23.910111+0000","last_active":"2024-03-26T21:32:35.117723+0000","last_peered":"2024-03-26T21:32:35.117723+0000","last_clean":"2024-03-26T21:32:35.117723+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:32:35.117723+0000","last_undegraded":"2024-03-26T21:32:35.117723+0000","last_fullsized":"2024-03-26T21:32:35.117723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.909981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.909981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:38.337727+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118440+0000","last_change":"2024-03-26T21:31:14.634077+0000","last_active":"2024-03-26T21:32:35.118440+0000","last_peered":"2024-03-26T21:32:35.118440+0000","last_clean":"2024-03-26T21:32:35.118440+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:32:35.118440+0000","last_undegraded":"2024-03-26T21:32:35.118440+0000","last_fullsized":"2024-03-26T21:32:35.118440+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:31:14.633936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.633936+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.129312+0000","last_change":"2024-03-26T21:30:31.739981+0000","last_active":"2024-03-26T21:32:35.129312+0000","last_peered":"2024-03-26T21:32:35.129312+0000","last_clean":"2024-03-26T21:30:19.265055+0000","last_became_active":"2024-03-26T21:30:31.739981+0000","last_became_peered":"2024-03-26T21:30:31.739981+0000","last_unstale":"2024-03-26T21:32:35.129312+0000","last_undegraded":"2024-03-26T21:32:35.129312+0000","last_fullsized":"2024-03-26T21:30:31.736585+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.958658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.958658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.124084+0000","last_change":"2024-03-26T21:32:23.286349+0000","last_active":"2024-03-26T21:32:35.124084+0000","last_peered":"2024-03-26T21:32:35.124084+0000","last_clean":"2024-03-26T21:32:35.124084+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:32:35.124084+0000","last_undegraded":"2024-03-26T21:32:35.124084+0000","last_fullsized":"2024-03-26T21:32:35.124084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.286238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.286238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:43.657258+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127668+0000","last_change":"2024-03-26T21:30:31.742874+0000","last_active":"2024-03-26T21:32:35.127668+0000","last_peered":"2024-03-26T21:32:35.127668+0000","last_clean":"2024-03-26T21:30:30.545271+0000","last_became_active":"2024-03-26T21:30:31.742874+0000","last_became_peered":"2024-03-26T21:30:31.742874+0000","last_unstale":"2024-03-26T21:32:35.127668+0000","last_undegraded":"2024-03-26T21:32:35.127668+0000","last_fullsized":"2024-03-26T21:30:31.735756+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:30.545190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:30.545190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.790611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.122595+0000","last_change":"2024-03-26T21:31:38.685593+0000","last_active":"2024-03-26T21:32:35.122595+0000","last_peered":"2024-03-26T21:32:35.122595+0000","last_clean":"2024-03-26T21:32:35.122595+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:32:35.122595+0000","last_undegraded":"2024-03-26T21:32:35.122595+0000","last_fullsized":"2024-03-26T21:32:35.122595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.685453+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.982628+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127896+0000","last_change":"2024-03-26T21:30:31.742256+0000","last_active":"2024-03-26T21:32:35.127896+0000","last_peered":"2024-03-26T21:32:35.127896+0000","last_clean":"2024-03-26T21:30:27.542711+0000","last_became_active":"2024-03-26T21:30:31.742256+0000","last_became_peered":"2024-03-26T21:30:31.742256+0000","last_unstale":"2024-03-26T21:32:35.127896+0000","last_undegraded":"2024-03-26T21:32:35.127896+0000","last_fullsized":"2024-03-26T21:30:31.733298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:30:27.542602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.790647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118550+0000","last_change":"2024-03-26T21:31:27.662639+0000","last_active":"2024-03-26T21:32:35.118550+0000","last_peered":"2024-03-26T21:32:35.118550+0000","last_clean":"2024-03-26T21:32:35.118550+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:32:35.118550+0000","last_undegraded":"2024-03-26T21:32:35.118550+0000","last_fullsized":"2024-03-26T21:32:35.118550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:27.662555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:27.662555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.105293+0000","last_change":"2024-03-26T21:31:31.675402+0000","last_active":"2024-03-26T21:32:35.105293+0000","last_peered":"2024-03-26T21:32:35.105293+0000","last_clean":"2024-03-26T21:32:35.105293+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:32:35.105293+0000","last_undegraded":"2024-03-26T21:32:35.105293+0000","last_fullsized":"2024-03-26T21:32:35.105293+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:31.675298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:31.675298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:55.883626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":165,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120333+0000","last_change":"2024-03-26T21:32:18.220547+0000","last_active":"2024-03-26T21:32:35.120333+0000","last_peered":"2024-03-26T21:32:35.120333+0000","last_clean":"2024-03-26T21:32:35.120333+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:32:35.120333+0000","last_undegraded":"2024-03-26T21:32:35.120333+0000","last_fullsized":"2024-03-26T21:32:35.120333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:32:18.214216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.214216+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:47.711041+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"126'8206","reported_seq":8357,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.421941+0000","last_change":"2024-03-26T21:32:20.035532+0000","last_active":"2024-03-26T21:32:36.421941+0000","last_peered":"2024-03-26T21:32:36.421941+0000","last_clean":"2024-03-26T21:32:36.421941+0000","last_became_active":"2024-03-26T21:30:38.543645+0000","last_became_peered":"2024-03-26T21:30:38.543645+0000","last_unstale":"2024-03-26T21:32:36.421941+0000","last_undegraded":"2024-03-26T21:32:36.421941+0000","last_fullsized":"2024-03-26T21:32:36.421941+0000","mapping_epoch":95,"log_start":"106'3100","ondisk_log_start":"106'3100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'7201","last_scrub_stamp":"2024-03-26T21:32:20.035053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:20.035053+0000","objects_scrubbed":7160,"log_size":5106,"log_dups_size":0,"ondisk_log_size":5106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:21.343165+0000","scrub_duration":2101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430833664,"num_objects":6574,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":472960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118015+0000","last_change":"2024-03-26T21:32:30.548012+0000","last_active":"2024-03-26T21:32:35.118015+0000","last_peered":"2024-03-26T21:32:35.118015+0000","last_clean":"2024-03-26T21:32:35.118015+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:32:35.118015+0000","last_undegraded":"2024-03-26T21:32:35.118015+0000","last_fullsized":"2024-03-26T21:32:35.118015+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.547900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:38.355082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":151,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.129063+0000","last_change":"2024-03-26T21:30:31.738782+0000","last_active":"2024-03-26T21:32:35.129063+0000","last_peered":"2024-03-26T21:32:35.129063+0000","last_clean":"2024-03-26T21:30:19.145088+0000","last_became_active":"2024-03-26T21:30:31.738782+0000","last_became_peered":"2024-03-26T21:30:31.738782+0000","last_unstale":"2024-03-26T21:32:35.129063+0000","last_undegraded":"2024-03-26T21:30:31.736341+0000","last_fullsized":"2024-03-26T21:30:31.736205+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":167,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120261+0000","last_change":"2024-03-26T21:32:16.596402+0000","last_active":"2024-03-26T21:32:35.120261+0000","last_peered":"2024-03-26T21:32:35.120261+0000","last_clean":"2024-03-26T21:32:35.120261+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:32:35.120261+0000","last_undegraded":"2024-03-26T21:32:35.120261+0000","last_fullsized":"2024-03-26T21:32:35.120261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:16.596266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:16.596266+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:35.953434+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":183,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.127967+0000","last_change":"2024-03-26T21:30:31.743316+0000","last_active":"2024-03-26T21:32:35.127967+0000","last_peered":"2024-03-26T21:32:35.127967+0000","last_clean":"2024-03-26T21:30:29.906838+0000","last_became_active":"2024-03-26T21:30:31.743316+0000","last_became_peered":"2024-03-26T21:30:31.743316+0000","last_unstale":"2024-03-26T21:32:35.127967+0000","last_undegraded":"2024-03-26T21:30:31.736202+0000","last_fullsized":"2024-03-26T21:30:31.736088+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"126'8226","reported_seq":8381,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.553520+0000","last_change":"2024-03-26T21:32:22.631630+0000","last_active":"2024-03-26T21:32:36.553520+0000","last_peered":"2024-03-26T21:32:36.553520+0000","last_clean":"2024-03-26T21:32:36.553520+0000","last_became_active":"2024-03-26T21:30:38.548054+0000","last_became_peered":"2024-03-26T21:30:38.548054+0000","last_unstale":"2024-03-26T21:32:36.553520+0000","last_undegraded":"2024-03-26T21:32:36.553520+0000","last_fullsized":"2024-03-26T21:32:36.553520+0000","mapping_epoch":95,"log_start":"108'3600","ondisk_log_start":"108'3600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7393","last_scrub_stamp":"2024-03-26T21:32:22.631335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.631335+0000","objects_scrubbed":7321,"log_size":4626,"log_dups_size":0,"ondisk_log_size":4626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:49.228273+0000","scrub_duration":3390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8226,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118690+0000","last_change":"2024-03-26T21:30:40.780018+0000","last_active":"2024-03-26T21:32:35.118690+0000","last_peered":"2024-03-26T21:32:35.118690+0000","last_clean":"2024-03-26T21:32:35.118690+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:32:35.118690+0000","last_undegraded":"2024-03-26T21:32:35.118690+0000","last_fullsized":"2024-03-26T21:32:35.118690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:40.779910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:40.779910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":148,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.128485+0000","last_change":"2024-03-26T21:30:31.744165+0000","last_active":"2024-03-26T21:32:35.128485+0000","last_peered":"2024-03-26T21:32:35.128485+0000","last_clean":"2024-03-26T21:30:19.145058+0000","last_became_active":"2024-03-26T21:30:31.744165+0000","last_became_peered":"2024-03-26T21:30:31.744165+0000","last_unstale":"2024-03-26T21:32:35.128485+0000","last_undegraded":"2024-03-26T21:30:31.735909+0000","last_fullsized":"2024-03-26T21:30:31.735789+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:29:44.612711+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.735120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":115,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.117005+0000","last_change":"2024-03-26T21:30:31.740300+0000","last_active":"2024-03-26T21:32:35.117005+0000","last_peered":"2024-03-26T21:32:35.117005+0000","last_clean":"2024-03-26T21:28:42.051440+0000","last_became_active":"2024-03-26T21:30:31.740300+0000","last_became_peered":"2024-03-26T21:30:31.740300+0000","last_unstale":"2024-03-26T21:32:35.117005+0000","last_undegraded":"2024-03-26T21:30:31.730730+0000","last_fullsized":"2024-03-26T21:30:31.730592+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:29:42.608450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.608450+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128297+0000","last_change":"2024-03-26T21:31:33.234806+0000","last_active":"2024-03-26T21:32:35.128297+0000","last_peered":"2024-03-26T21:32:35.128297+0000","last_clean":"2024-03-26T21:32:35.128297+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:32:35.128297+0000","last_undegraded":"2024-03-26T21:32:35.128297+0000","last_fullsized":"2024-03-26T21:32:35.128297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:33.234681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:33.234681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.001485+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"126'8508","reported_seq":8577,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:36.528142+0000","last_change":"2024-03-26T21:30:53.719400+0000","last_active":"2024-03-26T21:32:36.528142+0000","last_peered":"2024-03-26T21:32:36.528142+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.546367+0000","last_became_peered":"2024-03-26T21:30:38.546367+0000","last_unstale":"2024-03-26T21:32:36.528142+0000","last_undegraded":"2024-03-26T21:30:53.709328+0000","last_fullsized":"2024-03-26T21:30:38.539071+0000","mapping_epoch":95,"log_start":"106'3018","ondisk_log_start":"106'3018","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442892288,"num_objects":6758,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6758}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.127502+0000","last_change":"2024-03-26T21:32:18.784739+0000","last_active":"2024-03-26T21:32:35.127502+0000","last_peered":"2024-03-26T21:32:35.127502+0000","last_clean":"2024-03-26T21:32:35.127502+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:32:35.127502+0000","last_undegraded":"2024-03-26T21:32:35.127502+0000","last_fullsized":"2024-03-26T21:32:35.127502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:18.784626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.784626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:22.746354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.139724+0000","last_change":"2024-03-26T21:31:30.180324+0000","last_active":"2024-03-26T21:32:35.139724+0000","last_peered":"2024-03-26T21:32:35.139724+0000","last_clean":"2024-03-26T21:32:35.139724+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:32:35.139724+0000","last_undegraded":"2024-03-26T21:32:35.139724+0000","last_fullsized":"2024-03-26T21:32:35.139724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:30.180213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.180213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.156578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128417+0000","last_change":"2024-03-26T21:31:16.725440+0000","last_active":"2024-03-26T21:32:35.128417+0000","last_peered":"2024-03-26T21:32:35.128417+0000","last_clean":"2024-03-26T21:32:35.128417+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:32:35.128417+0000","last_undegraded":"2024-03-26T21:32:35.128417+0000","last_fullsized":"2024-03-26T21:32:35.128417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:16.725334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:16.725334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.801808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":182,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.106003+0000","last_change":"2024-03-26T21:31:28.640751+0000","last_active":"2024-03-26T21:32:35.106003+0000","last_peered":"2024-03-26T21:32:35.106003+0000","last_clean":"2024-03-26T21:32:35.106003+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:32:35.106003+0000","last_undegraded":"2024-03-26T21:32:35.106003+0000","last_fullsized":"2024-03-26T21:32:35.106003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:31:28.640572+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:31:28.640572+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.640572+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.382562+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"126'8419","reported_seq":8488,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:36.445179+0000","last_change":"2024-03-26T21:30:53.710423+0000","last_active":"2024-03-26T21:32:36.445179+0000","last_peered":"2024-03-26T21:32:36.445179+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.540186+0000","last_became_peered":"2024-03-26T21:30:38.540186+0000","last_unstale":"2024-03-26T21:32:36.445179+0000","last_undegraded":"2024-03-26T21:30:53.707983+0000","last_fullsized":"2024-03-26T21:30:38.534077+0000","mapping_epoch":95,"log_start":"102'1400","ondisk_log_start":"102'1400","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":7019,"log_dups_size":0,"ondisk_log_size":7019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437321728,"num_objects":6673,"num_object_clones":0,"num_object_copies":20019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":6673}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118221+0000","last_change":"2024-03-26T21:31:37.722079+0000","last_active":"2024-03-26T21:32:35.118221+0000","last_peered":"2024-03-26T21:32:35.118221+0000","last_clean":"2024-03-26T21:32:35.118221+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:32:35.118221+0000","last_undegraded":"2024-03-26T21:32:35.118221+0000","last_fullsized":"2024-03-26T21:32:35.118221+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.721821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:57.948602+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.138512+0000","last_change":"2024-03-26T21:30:31.741930+0000","last_active":"2024-03-26T21:32:35.138512+0000","last_peered":"2024-03-26T21:32:35.138512+0000","last_clean":"2024-03-26T21:30:19.264875+0000","last_became_active":"2024-03-26T21:30:31.741930+0000","last_became_peered":"2024-03-26T21:30:31.741930+0000","last_unstale":"2024-03-26T21:32:35.138512+0000","last_undegraded":"2024-03-26T21:32:35.138512+0000","last_fullsized":"2024-03-26T21:30:31.734873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.738632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.738632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118050+0000","last_change":"2024-03-26T21:31:39.585713+0000","last_active":"2024-03-26T21:32:35.118050+0000","last_peered":"2024-03-26T21:32:35.118050+0000","last_clean":"2024-03-26T21:32:35.118050+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:32:35.118050+0000","last_undegraded":"2024-03-26T21:32:35.118050+0000","last_fullsized":"2024-03-26T21:32:35.118050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.585631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.992136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":145,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.118772+0000","last_change":"2024-03-26T21:30:31.747569+0000","last_active":"2024-03-26T21:32:35.118772+0000","last_peered":"2024-03-26T21:32:35.118772+0000","last_clean":"2024-03-26T21:30:19.056675+0000","last_became_active":"2024-03-26T21:30:31.747569+0000","last_became_peered":"2024-03-26T21:30:31.747569+0000","last_unstale":"2024-03-26T21:32:35.118772+0000","last_undegraded":"2024-03-26T21:30:31.738166+0000","last_fullsized":"2024-03-26T21:30:31.738045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:28:35.824090+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"126'8423","reported_seq":8577,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.270292+0000","last_change":"2024-03-26T21:32:16.952114+0000","last_active":"2024-03-26T21:32:36.270292+0000","last_peered":"2024-03-26T21:32:36.270292+0000","last_clean":"2024-03-26T21:32:36.270292+0000","last_became_active":"2024-03-26T21:30:38.548880+0000","last_became_peered":"2024-03-26T21:30:38.548880+0000","last_unstale":"2024-03-26T21:32:36.270292+0000","last_undegraded":"2024-03-26T21:32:36.270292+0000","last_fullsized":"2024-03-26T21:32:36.270292+0000","mapping_epoch":95,"log_start":"103'2212","ondisk_log_start":"103'2212","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7249","last_scrub_stamp":"2024-03-26T21:32:16.951894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:16.951894+0000","objects_scrubbed":7162,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:42.888215+0000","scrub_duration":2290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441122816,"num_objects":6731,"num_object_clones":0,"num_object_copies":20193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":484928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.119822+0000","last_change":"2024-03-26T21:30:31.738178+0000","last_active":"2024-03-26T21:32:35.119822+0000","last_peered":"2024-03-26T21:32:35.119822+0000","last_clean":"2024-03-26T21:30:19.792003+0000","last_became_active":"2024-03-26T21:30:31.738178+0000","last_became_peered":"2024-03-26T21:30:31.738178+0000","last_unstale":"2024-03-26T21:32:35.119822+0000","last_undegraded":"2024-03-26T21:32:35.119822+0000","last_fullsized":"2024-03-26T21:30:31.731810+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:29:42.726312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117766+0000","last_change":"2024-03-26T21:32:15.590737+0000","last_active":"2024-03-26T21:32:35.117766+0000","last_peered":"2024-03-26T21:32:35.117766+0000","last_clean":"2024-03-26T21:32:35.117766+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:32:35.117766+0000","last_undegraded":"2024-03-26T21:32:35.117766+0000","last_fullsized":"2024-03-26T21:32:35.117766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:32:15.590617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:15.590617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:15.765854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":75,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118321+0000","last_change":"2024-03-26T21:30:31.740606+0000","last_active":"2024-03-26T21:32:35.118321+0000","last_peered":"2024-03-26T21:32:35.118321+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.740606+0000","last_became_peered":"2024-03-26T21:30:31.740606+0000","last_unstale":"2024-03-26T21:32:35.118321+0000","last_undegraded":"2024-03-26T21:32:35.118321+0000","last_fullsized":"2024-03-26T21:30:31.737530+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:15.290476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:15.290476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":193,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117510+0000","last_change":"2024-03-26T21:31:30.659923+0000","last_active":"2024-03-26T21:32:35.117510+0000","last_peered":"2024-03-26T21:32:35.117510+0000","last_clean":"2024-03-26T21:32:35.117510+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:32:35.117510+0000","last_undegraded":"2024-03-26T21:32:35.117510+0000","last_fullsized":"2024-03-26T21:32:35.117510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:31:30.659832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.659832+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.256066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.117363+0000","last_change":"2024-03-26T21:30:31.745830+0000","last_active":"2024-03-26T21:32:35.117363+0000","last_peered":"2024-03-26T21:32:35.117363+0000","last_clean":"2024-03-26T21:30:19.056424+0000","last_became_active":"2024-03-26T21:30:31.745830+0000","last_became_peered":"2024-03-26T21:30:31.745830+0000","last_unstale":"2024-03-26T21:32:35.117363+0000","last_undegraded":"2024-03-26T21:32:35.117363+0000","last_fullsized":"2024-03-26T21:30:31.736422+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:06.721880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:06.721880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"126'8301","reported_seq":8370,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:36.472883+0000","last_change":"2024-03-26T21:30:53.720906+0000","last_active":"2024-03-26T21:32:36.472883+0000","last_peered":"2024-03-26T21:32:36.472883+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.544117+0000","last_became_peered":"2024-03-26T21:30:38.544117+0000","last_unstale":"2024-03-26T21:32:36.472883+0000","last_undegraded":"2024-03-26T21:30:53.719381+0000","last_fullsized":"2024-03-26T21:30:38.538732+0000","mapping_epoch":95,"log_start":"105'2768","ondisk_log_start":"105'2768","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5533,"log_dups_size":0,"ondisk_log_size":5533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437714944,"num_objects":6679,"num_object_clones":0,"num_object_copies":20037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6679}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128581+0000","last_change":"2024-03-26T21:31:36.751872+0000","last_active":"2024-03-26T21:32:35.128581+0000","last_peered":"2024-03-26T21:32:35.128581+0000","last_clean":"2024-03-26T21:32:35.128581+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:32:35.128581+0000","last_undegraded":"2024-03-26T21:32:35.128581+0000","last_fullsized":"2024-03-26T21:32:35.128581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:31:36.751794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.345858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":154,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.121682+0000","last_change":"2024-03-26T21:32:25.911544+0000","last_active":"2024-03-26T21:32:35.121682+0000","last_peered":"2024-03-26T21:32:35.121682+0000","last_clean":"2024-03-26T21:32:35.121682+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:32:35.121682+0000","last_undegraded":"2024-03-26T21:32:35.121682+0000","last_fullsized":"2024-03-26T21:32:35.121682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:25.911437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:25.911437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:48.343466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.127767+0000","last_change":"2024-03-26T21:31:28.109942+0000","last_active":"2024-03-26T21:32:35.127767+0000","last_peered":"2024-03-26T21:32:35.127767+0000","last_clean":"2024-03-26T21:32:35.127767+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:32:35.127767+0000","last_undegraded":"2024-03-26T21:32:35.127767+0000","last_fullsized":"2024-03-26T21:32:35.127767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.109786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.109786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:46.736255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117851+0000","last_change":"2024-03-26T21:31:28.105503+0000","last_active":"2024-03-26T21:32:35.117851+0000","last_peered":"2024-03-26T21:32:35.117851+0000","last_clean":"2024-03-26T21:32:35.117851+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:32:35.117851+0000","last_undegraded":"2024-03-26T21:32:35.117851+0000","last_fullsized":"2024-03-26T21:32:35.117851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:31:28.105376+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.105376+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:48.590370+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128099+0000","last_change":"2024-03-26T21:31:25.089539+0000","last_active":"2024-03-26T21:32:35.128099+0000","last_peered":"2024-03-26T21:32:35.128099+0000","last_clean":"2024-03-26T21:32:35.128099+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:32:35.128099+0000","last_undegraded":"2024-03-26T21:32:35.128099+0000","last_fullsized":"2024-03-26T21:32:35.128099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:25.089066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.089066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:52.881598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"126'8239","reported_seq":8393,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.508684+0000","last_change":"2024-03-26T21:32:22.439431+0000","last_active":"2024-03-26T21:32:36.508684+0000","last_peered":"2024-03-26T21:32:36.508684+0000","last_clean":"2024-03-26T21:32:36.508684+0000","last_became_active":"2024-03-26T21:30:38.550658+0000","last_became_peered":"2024-03-26T21:30:38.550658+0000","last_unstale":"2024-03-26T21:32:36.508684+0000","last_undegraded":"2024-03-26T21:32:36.508684+0000","last_fullsized":"2024-03-26T21:32:36.508684+0000","mapping_epoch":95,"log_start":"105'2716","ondisk_log_start":"105'2716","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7328","last_scrub_stamp":"2024-03-26T21:32:22.439030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.439030+0000","objects_scrubbed":7259,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:39.860496+0000","scrub_duration":3392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432209920,"num_objects":6595,"num_object_clones":0,"num_object_copies":19785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":474688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118442+0000","last_change":"2024-03-26T21:30:31.744917+0000","last_active":"2024-03-26T21:32:35.118442+0000","last_peered":"2024-03-26T21:32:35.118442+0000","last_clean":"2024-03-26T21:30:19.907127+0000","last_became_active":"2024-03-26T21:30:31.744917+0000","last_became_peered":"2024-03-26T21:30:31.744917+0000","last_unstale":"2024-03-26T21:32:35.118442+0000","last_undegraded":"2024-03-26T21:32:35.118442+0000","last_fullsized":"2024-03-26T21:30:31.734562+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:08.472985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:08.472985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":399,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.127623+0000","last_change":"2024-03-26T21:32:19.758672+0000","last_active":"2024-03-26T21:32:35.127623+0000","last_peered":"2024-03-26T21:32:35.127623+0000","last_clean":"2024-03-26T21:32:35.127623+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:32:35.127623+0000","last_undegraded":"2024-03-26T21:32:35.127623+0000","last_fullsized":"2024-03-26T21:32:35.127623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:32:19.758532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:32:19.758532+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:27.240492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.142877+0000","last_change":"2024-03-26T21:31:14.873351+0000","last_active":"2024-03-26T21:32:35.142877+0000","last_peered":"2024-03-26T21:32:35.142877+0000","last_clean":"2024-03-26T21:32:35.142877+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:32:35.142877+0000","last_undegraded":"2024-03-26T21:32:35.142877+0000","last_fullsized":"2024-03-26T21:32:35.142877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:14.872965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:14.872965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.836988+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"94'25","reported_seq":217,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.105928+0000","last_change":"2024-03-26T21:30:31.737537+0000","last_active":"2024-03-26T21:32:35.105928+0000","last_peered":"2024-03-26T21:32:35.105928+0000","last_clean":"2024-03-26T21:30:29.907049+0000","last_became_active":"2024-03-26T21:30:31.737537+0000","last_became_peered":"2024-03-26T21:30:31.737537+0000","last_unstale":"2024-03-26T21:32:35.105928+0000","last_undegraded":"2024-03-26T21:30:31.733102+0000","last_fullsized":"2024-03-26T21:30:31.732979+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'24","last_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:29:55.903057+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128326+0000","last_change":"2024-03-26T21:32:22.778381+0000","last_active":"2024-03-26T21:32:35.128326+0000","last_peered":"2024-03-26T21:32:35.128326+0000","last_clean":"2024-03-26T21:32:35.128326+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:32:35.128326+0000","last_undegraded":"2024-03-26T21:32:35.128326+0000","last_fullsized":"2024-03-26T21:32:35.128326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:22.778299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.778299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:29.419969+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"126'8227","reported_seq":8378,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.271498+0000","last_change":"2024-03-26T21:32:18.588096+0000","last_active":"2024-03-26T21:32:36.271498+0000","last_peered":"2024-03-26T21:32:36.271498+0000","last_clean":"2024-03-26T21:32:36.271498+0000","last_became_active":"2024-03-26T21:30:38.543531+0000","last_became_peered":"2024-03-26T21:30:38.543531+0000","last_unstale":"2024-03-26T21:32:36.271498+0000","last_undegraded":"2024-03-26T21:32:36.271498+0000","last_fullsized":"2024-03-26T21:32:36.271498+0000","mapping_epoch":95,"log_start":"106'2900","ondisk_log_start":"106'2900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7176","last_scrub_stamp":"2024-03-26T21:32:18.587792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.587792+0000","objects_scrubbed":7129,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:27.118278+0000","scrub_duration":1845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432078848,"num_objects":6593,"num_object_clones":0,"num_object_copies":19779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":474240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116625+0000","last_change":"2024-03-26T21:31:39.079125+0000","last_active":"2024-03-26T21:32:35.116625+0000","last_peered":"2024-03-26T21:32:35.116625+0000","last_clean":"2024-03-26T21:32:35.116625+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:32:35.116625+0000","last_undegraded":"2024-03-26T21:32:35.116625+0000","last_fullsized":"2024-03-26T21:32:35.116625+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.079005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.079005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:03.373633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118424+0000","last_change":"2024-03-26T21:30:31.743896+0000","last_active":"2024-03-26T21:32:35.118424+0000","last_peered":"2024-03-26T21:32:35.118424+0000","last_clean":"2024-03-26T21:30:19.055870+0000","last_became_active":"2024-03-26T21:30:31.743896+0000","last_became_peered":"2024-03-26T21:30:31.743896+0000","last_unstale":"2024-03-26T21:32:35.118424+0000","last_undegraded":"2024-03-26T21:32:35.118424+0000","last_fullsized":"2024-03-26T21:30:31.733630+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:30:17.708355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":392,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.654489+0000","last_change":"2024-03-26T21:31:29.792073+0000","last_active":"2024-03-26T21:32:35.654489+0000","last_peered":"2024-03-26T21:32:35.654489+0000","last_clean":"2024-03-26T21:32:35.654489+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:32:35.654489+0000","last_undegraded":"2024-03-26T21:32:35.654489+0000","last_fullsized":"2024-03-26T21:32:35.654489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:31:29.791964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.791964+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:47.167445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":180,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120870+0000","last_change":"2024-03-26T21:31:23.770810+0000","last_active":"2024-03-26T21:32:35.120870+0000","last_peered":"2024-03-26T21:32:35.120870+0000","last_clean":"2024-03-26T21:32:35.120870+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:32:35.120870+0000","last_undegraded":"2024-03-26T21:32:35.120870+0000","last_fullsized":"2024-03-26T21:32:35.120870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:31:23.770682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:23.770682+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:37.819857+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118703+0000","last_change":"2024-03-26T21:32:28.548229+0000","last_active":"2024-03-26T21:32:35.118703+0000","last_peered":"2024-03-26T21:32:35.118703+0000","last_clean":"2024-03-26T21:32:35.118703+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:32:35.118703+0000","last_undegraded":"2024-03-26T21:32:35.118703+0000","last_fullsized":"2024-03-26T21:32:35.118703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:28.548115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:28.548115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:54.970716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"126'8266","reported_seq":8414,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.271793+0000","last_change":"2024-03-26T21:32:36.082531+0000","last_active":"2024-03-26T21:32:36.271793+0000","last_peered":"2024-03-26T21:32:36.271793+0000","last_clean":"2024-03-26T21:32:36.271793+0000","last_became_active":"2024-03-26T21:30:38.543747+0000","last_became_peered":"2024-03-26T21:30:38.543747+0000","last_unstale":"2024-03-26T21:32:36.271793+0000","last_undegraded":"2024-03-26T21:32:36.271793+0000","last_fullsized":"2024-03-26T21:32:36.271793+0000","mapping_epoch":95,"log_start":"104'2641","ondisk_log_start":"104'2641","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'8255","last_scrub_stamp":"2024-03-26T21:32:36.082442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.082442+0000","objects_scrubbed":6758,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:59.098085+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439484416,"num_objects":6706,"num_object_clones":0,"num_object_copies":20118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8266,"num_write_kb":479104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116626+0000","last_change":"2024-03-26T21:31:30.064302+0000","last_active":"2024-03-26T21:32:35.116626+0000","last_peered":"2024-03-26T21:32:35.116626+0000","last_clean":"2024-03-26T21:32:35.116626+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:32:35.116626+0000","last_undegraded":"2024-03-26T21:32:35.116626+0000","last_fullsized":"2024-03-26T21:32:35.116626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:30.064009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:30.064009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.180293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116904+0000","last_change":"2024-03-26T21:31:25.137611+0000","last_active":"2024-03-26T21:32:35.116904+0000","last_peered":"2024-03-26T21:32:35.116904+0000","last_clean":"2024-03-26T21:32:35.116904+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:32:35.116904+0000","last_undegraded":"2024-03-26T21:32:35.116904+0000","last_fullsized":"2024-03-26T21:32:35.116904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:25.137255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:31:25.137255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:45.125501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":178,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118610+0000","last_change":"2024-03-26T21:31:32.754170+0000","last_active":"2024-03-26T21:32:35.118610+0000","last_peered":"2024-03-26T21:32:35.118610+0000","last_clean":"2024-03-26T21:32:35.118610+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:32:35.118610+0000","last_undegraded":"2024-03-26T21:32:35.118610+0000","last_fullsized":"2024-03-26T21:32:35.118610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:31:32.754010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.754010+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:56.783769+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.106115+0000","last_change":"2024-03-26T21:31:21.554718+0000","last_active":"2024-03-26T21:32:35.106115+0000","last_peered":"2024-03-26T21:32:35.106115+0000","last_clean":"2024-03-26T21:32:35.106115+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:32:35.106115+0000","last_undegraded":"2024-03-26T21:32:35.106115+0000","last_fullsized":"2024-03-26T21:32:35.106115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:21.554604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:21.554604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:40.440766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"126'8174","reported_seq":8243,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:36.555213+0000","last_change":"2024-03-26T21:30:53.715187+0000","last_active":"2024-03-26T21:32:36.555213+0000","last_peered":"2024-03-26T21:32:36.555213+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.547364+0000","last_became_peered":"2024-03-26T21:30:38.547364+0000","last_unstale":"2024-03-26T21:32:36.555213+0000","last_undegraded":"2024-03-26T21:30:53.713329+0000","last_fullsized":"2024-03-26T21:30:38.537882+0000","mapping_epoch":95,"log_start":"108'3500","ondisk_log_start":"108'3500","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.299657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430571520,"num_objects":6570,"num_object_clones":0,"num_object_copies":19710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8174,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":6570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.116979+0000","last_change":"2024-03-26T21:32:13.010288+0000","last_active":"2024-03-26T21:32:35.116979+0000","last_peered":"2024-03-26T21:32:35.116979+0000","last_clean":"2024-03-26T21:32:35.116979+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:32:35.116979+0000","last_undegraded":"2024-03-26T21:32:35.116979+0000","last_fullsized":"2024-03-26T21:32:35.116979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:13.010170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.010170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:33.926303+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":154,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.525364+0000","last_change":"2024-03-26T21:32:35.525387+0000","last_active":"2024-03-26T21:32:35.525364+0000","last_peered":"2024-03-26T21:32:35.525364+0000","last_clean":"2024-03-26T21:32:35.525364+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:32:35.525364+0000","last_undegraded":"2024-03-26T21:32:35.525364+0000","last_fullsized":"2024-03-26T21:32:35.525364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:32:35.525298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:04.901329+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":152,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.122656+0000","last_change":"2024-03-26T21:30:31.739773+0000","last_active":"2024-03-26T21:32:35.122656+0000","last_peered":"2024-03-26T21:32:35.122656+0000","last_clean":"2024-03-26T21:30:19.056603+0000","last_became_active":"2024-03-26T21:30:31.739773+0000","last_became_peered":"2024-03-26T21:30:31.739773+0000","last_unstale":"2024-03-26T21:32:35.122656+0000","last_undegraded":"2024-03-26T21:30:31.733621+0000","last_fullsized":"2024-03-26T21:30:31.733509+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:30:02.076064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:02.076064+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":195,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.118110+0000","last_change":"2024-03-26T21:32:11.923454+0000","last_active":"2024-03-26T21:32:35.118110+0000","last_peered":"2024-03-26T21:32:35.118110+0000","last_clean":"2024-03-26T21:32:35.118110+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:32:35.118110+0000","last_undegraded":"2024-03-26T21:32:35.118110+0000","last_fullsized":"2024-03-26T21:32:35.118110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:11.916784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:11.916784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:29.585677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"126'8258","reported_seq":8388,"reported_epoch":126,"state":"active+clean+scrubbing","last_fresh":"2024-03-26T21:32:36.571976+0000","last_change":"2024-03-26T21:32:34.296785+0000","last_active":"2024-03-26T21:32:36.571976+0000","last_peered":"2024-03-26T21:32:36.571976+0000","last_clean":"2024-03-26T21:32:36.571976+0000","last_became_active":"2024-03-26T21:30:38.549806+0000","last_became_peered":"2024-03-26T21:30:38.549806+0000","last_unstale":"2024-03-26T21:32:36.571976+0000","last_undegraded":"2024-03-26T21:32:36.571976+0000","last_fullsized":"2024-03-26T21:32:36.571976+0000","mapping_epoch":95,"log_start":"108'3600","ondisk_log_start":"108'3600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":5349,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":474240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128194+0000","last_change":"2024-03-26T21:32:10.732813+0000","last_active":"2024-03-26T21:32:35.128194+0000","last_peered":"2024-03-26T21:32:35.128194+0000","last_clean":"2024-03-26T21:32:35.128194+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:32:35.128194+0000","last_undegraded":"2024-03-26T21:32:35.128194+0000","last_fullsized":"2024-03-26T21:32:35.128194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:10.732733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:10.732733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:20.387147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":134,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.120973+0000","last_change":"2024-03-26T21:30:31.746821+0000","last_active":"2024-03-26T21:32:35.120973+0000","last_peered":"2024-03-26T21:32:35.120973+0000","last_clean":"2024-03-26T21:30:19.056856+0000","last_became_active":"2024-03-26T21:30:31.746821+0000","last_became_peered":"2024-03-26T21:30:31.746821+0000","last_unstale":"2024-03-26T21:32:35.120973+0000","last_undegraded":"2024-03-26T21:30:31.736673+0000","last_fullsized":"2024-03-26T21:30:31.736528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:36.898078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":114,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.117383+0000","last_change":"2024-03-26T21:30:31.739394+0000","last_active":"2024-03-26T21:32:35.117383+0000","last_peered":"2024-03-26T21:32:35.117383+0000","last_clean":"2024-03-26T21:28:42.253768+0000","last_became_active":"2024-03-26T21:30:31.739394+0000","last_became_peered":"2024-03-26T21:30:31.739394+0000","last_unstale":"2024-03-26T21:32:35.117383+0000","last_undegraded":"2024-03-26T21:30:31.732454+0000","last_fullsized":"2024-03-26T21:30:31.732332+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"94'9","reported_seq":191,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.120708+0000","last_change":"2024-03-26T21:30:31.745653+0000","last_active":"2024-03-26T21:32:35.120708+0000","last_peered":"2024-03-26T21:32:35.120708+0000","last_clean":"2024-03-26T21:30:29.906563+0000","last_became_active":"2024-03-26T21:30:31.745653+0000","last_became_peered":"2024-03-26T21:30:31.745653+0000","last_unstale":"2024-03-26T21:32:35.120708+0000","last_undegraded":"2024-03-26T21:30:31.730030+0000","last_fullsized":"2024-03-26T21:30:31.729924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:29:59.694434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:59.694434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"126'8303","reported_seq":8372,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:36.573670+0000","last_change":"2024-03-26T21:30:53.715233+0000","last_active":"2024-03-26T21:32:36.573670+0000","last_peered":"2024-03-26T21:32:36.573670+0000","last_clean":"2024-03-26T21:30:37.495090+0000","last_became_active":"2024-03-26T21:30:38.548505+0000","last_became_peered":"2024-03-26T21:30:38.548505+0000","last_unstale":"2024-03-26T21:32:36.573670+0000","last_undegraded":"2024-03-26T21:30:53.710995+0000","last_fullsized":"2024-03-26T21:30:38.537548+0000","mapping_epoch":95,"log_start":"108'3600","ondisk_log_start":"108'3600","created":95,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:38.325719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434831360,"num_objects":6635,"num_object_clones":0,"num_object_copies":19905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6635,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":6635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.143473+0000","last_change":"2024-03-26T21:30:31.738939+0000","last_active":"2024-03-26T21:32:35.143473+0000","last_peered":"2024-03-26T21:32:35.143473+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.738939+0000","last_became_peered":"2024-03-26T21:30:31.738939+0000","last_unstale":"2024-03-26T21:32:35.143473+0000","last_undegraded":"2024-03-26T21:32:35.143473+0000","last_fullsized":"2024-03-26T21:30:31.730286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:29:48.632355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.280946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120178+0000","last_change":"2024-03-26T21:32:17.198343+0000","last_active":"2024-03-26T21:32:35.120178+0000","last_peered":"2024-03-26T21:32:35.120178+0000","last_clean":"2024-03-26T21:32:35.120178+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:32:35.120178+0000","last_undegraded":"2024-03-26T21:32:35.120178+0000","last_fullsized":"2024-03-26T21:32:35.120178+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:17.198228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:17.198228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:34.541894+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120609+0000","last_change":"2024-03-26T21:32:33.337863+0000","last_active":"2024-03-26T21:32:35.120609+0000","last_peered":"2024-03-26T21:32:35.120609+0000","last_clean":"2024-03-26T21:32:35.120609+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:32:35.120609+0000","last_undegraded":"2024-03-26T21:32:35.120609+0000","last_fullsized":"2024-03-26T21:32:35.120609+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:32:33.337581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:33.337581+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:36.261185+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.119355+0000","last_change":"2024-03-26T21:30:31.738110+0000","last_active":"2024-03-26T21:32:35.119355+0000","last_peered":"2024-03-26T21:32:35.119355+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:30:31.738110+0000","last_became_peered":"2024-03-26T21:30:31.738110+0000","last_unstale":"2024-03-26T21:32:35.119355+0000","last_undegraded":"2024-03-26T21:32:35.119355+0000","last_fullsized":"2024-03-26T21:30:31.729956+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:41.280649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"126'8330","reported_seq":8482,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.289242+0000","last_change":"2024-03-26T21:32:28.144933+0000","last_active":"2024-03-26T21:32:36.289242+0000","last_peered":"2024-03-26T21:32:36.289242+0000","last_clean":"2024-03-26T21:32:36.289242+0000","last_became_active":"2024-03-26T21:30:38.539838+0000","last_became_peered":"2024-03-26T21:30:38.539838+0000","last_unstale":"2024-03-26T21:32:36.289242+0000","last_undegraded":"2024-03-26T21:32:36.289242+0000","last_fullsized":"2024-03-26T21:32:36.289242+0000","mapping_epoch":95,"log_start":"103'2160","ondisk_log_start":"103'2160","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'7791","last_scrub_stamp":"2024-03-26T21:32:28.144613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:28.144613+0000","objects_scrubbed":7281,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:47.542470+0000","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438042624,"num_objects":6684,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8330,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.117708+0000","last_change":"2024-03-26T21:31:34.602108+0000","last_active":"2024-03-26T21:32:35.117708+0000","last_peered":"2024-03-26T21:32:35.117708+0000","last_clean":"2024-03-26T21:32:35.117708+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:32:35.117708+0000","last_undegraded":"2024-03-26T21:32:35.117708+0000","last_fullsized":"2024-03-26T21:32:35.117708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:34.602001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:34.602001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:59.757795+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"94'9","reported_seq":124,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.118368+0000","last_change":"2024-03-26T21:30:31.738518+0000","last_active":"2024-03-26T21:32:35.118368+0000","last_peered":"2024-03-26T21:32:35.118368+0000","last_clean":"2024-03-26T21:28:42.046048+0000","last_became_active":"2024-03-26T21:30:31.738518+0000","last_became_peered":"2024-03-26T21:30:31.738518+0000","last_unstale":"2024-03-26T21:32:35.118368+0000","last_undegraded":"2024-03-26T21:30:31.734182+0000","last_fullsized":"2024-03-26T21:30:31.734060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.407216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.407216+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.120517+0000","last_change":"2024-03-26T21:31:29.955815+0000","last_active":"2024-03-26T21:32:35.120517+0000","last_peered":"2024-03-26T21:32:35.120517+0000","last_clean":"2024-03-26T21:32:35.120517+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:32:35.120517+0000","last_undegraded":"2024-03-26T21:32:35.120517+0000","last_fullsized":"2024-03-26T21:32:35.120517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:51.604999+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":162,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.119176+0000","last_change":"2024-03-26T21:30:31.746320+0000","last_active":"2024-03-26T21:32:35.119176+0000","last_peered":"2024-03-26T21:32:35.119176+0000","last_clean":"2024-03-26T21:30:19.057783+0000","last_became_active":"2024-03-26T21:30:31.746320+0000","last_became_peered":"2024-03-26T21:30:31.746320+0000","last_unstale":"2024-03-26T21:32:35.119176+0000","last_undegraded":"2024-03-26T21:30:31.733035+0000","last_fullsized":"2024-03-26T21:30:31.732940+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:30:11.086876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.086876+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"126'8193","reported_seq":8348,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.287730+0000","last_change":"2024-03-26T21:32:26.478537+0000","last_active":"2024-03-26T21:32:36.287730+0000","last_peered":"2024-03-26T21:32:36.287730+0000","last_clean":"2024-03-26T21:32:36.287730+0000","last_became_active":"2024-03-26T21:30:38.543981+0000","last_became_peered":"2024-03-26T21:30:38.543981+0000","last_unstale":"2024-03-26T21:32:36.287730+0000","last_undegraded":"2024-03-26T21:32:36.287730+0000","last_fullsized":"2024-03-26T21:32:36.287730+0000","mapping_epoch":95,"log_start":"103'2037","ondisk_log_start":"103'2037","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'7593","last_scrub_stamp":"2024-03-26T21:32:26.478332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.478332+0000","objects_scrubbed":7262,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:49.488306+0000","scrub_duration":1775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537624,"num_objects":6601,"num_object_clones":0,"num_object_copies":19803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8193,"num_write_kb":473345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.119248+0000","last_change":"2024-03-26T21:31:34.924180+0000","last_active":"2024-03-26T21:32:35.119248+0000","last_peered":"2024-03-26T21:32:35.119248+0000","last_clean":"2024-03-26T21:32:35.119248+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:32:35.119248+0000","last_undegraded":"2024-03-26T21:32:35.119248+0000","last_fullsized":"2024-03-26T21:32:35.119248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:31:34.924056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:49.555962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.118670+0000","last_change":"2024-03-26T21:30:31.742736+0000","last_active":"2024-03-26T21:32:35.118670+0000","last_peered":"2024-03-26T21:32:35.118670+0000","last_clean":"2024-03-26T21:30:19.907576+0000","last_became_active":"2024-03-26T21:30:31.742736+0000","last_became_peered":"2024-03-26T21:30:31.742736+0000","last_unstale":"2024-03-26T21:32:35.118670+0000","last_undegraded":"2024-03-26T21:32:35.118670+0000","last_fullsized":"2024-03-26T21:30:31.735097+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:28:37.828962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:39.507782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":74,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.127597+0000","last_change":"2024-03-26T21:30:31.738606+0000","last_active":"2024-03-26T21:32:35.127597+0000","last_peered":"2024-03-26T21:32:35.127597+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:30:31.738606+0000","last_became_peered":"2024-03-26T21:30:31.738606+0000","last_unstale":"2024-03-26T21:32:35.127597+0000","last_undegraded":"2024-03-26T21:32:35.127597+0000","last_fullsized":"2024-03-26T21:30:31.734264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:50.575609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:50.575609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":111,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.105855+0000","last_change":"2024-03-26T21:30:31.741438+0000","last_active":"2024-03-26T21:32:35.105855+0000","last_peered":"2024-03-26T21:32:35.105855+0000","last_clean":"2024-03-26T21:28:42.249232+0000","last_became_active":"2024-03-26T21:30:31.741438+0000","last_became_peered":"2024-03-26T21:30:31.741438+0000","last_unstale":"2024-03-26T21:32:35.105855+0000","last_undegraded":"2024-03-26T21:30:31.733158+0000","last_fullsized":"2024-03-26T21:30:31.733038+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"126'8320","reported_seq":8472,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.436166+0000","last_change":"2024-03-26T21:32:16.545631+0000","last_active":"2024-03-26T21:32:36.436166+0000","last_peered":"2024-03-26T21:32:36.436166+0000","last_clean":"2024-03-26T21:32:36.436166+0000","last_became_active":"2024-03-26T21:30:38.545485+0000","last_became_peered":"2024-03-26T21:30:38.545485+0000","last_unstale":"2024-03-26T21:32:36.436166+0000","last_undegraded":"2024-03-26T21:32:36.436166+0000","last_fullsized":"2024-03-26T21:32:36.436166+0000","mapping_epoch":95,"log_start":"102'1300","ondisk_log_start":"102'1300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7144","last_scrub_stamp":"2024-03-26T21:32:16.545400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:16.545400+0000","objects_scrubbed":7077,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:30.174618+0000","scrub_duration":2977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437649408,"num_objects":6678,"num_object_clones":0,"num_object_copies":20034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":479936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.129628+0000","last_change":"2024-03-26T21:30:31.741781+0000","last_active":"2024-03-26T21:32:35.129628+0000","last_peered":"2024-03-26T21:32:35.129628+0000","last_clean":"2024-03-26T21:30:19.265083+0000","last_became_active":"2024-03-26T21:30:31.741781+0000","last_became_peered":"2024-03-26T21:30:31.741781+0000","last_unstale":"2024-03-26T21:32:35.129628+0000","last_undegraded":"2024-03-26T21:32:35.129628+0000","last_fullsized":"2024-03-26T21:30:31.735785+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:30:04.923329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.117039+0000","last_change":"2024-03-26T21:30:31.743030+0000","last_active":"2024-03-26T21:32:35.117039+0000","last_peered":"2024-03-26T21:32:35.117039+0000","last_clean":"2024-03-26T21:30:19.057039+0000","last_became_active":"2024-03-26T21:30:31.743030+0000","last_became_peered":"2024-03-26T21:30:31.743030+0000","last_unstale":"2024-03-26T21:32:35.117039+0000","last_undegraded":"2024-03-26T21:32:35.117039+0000","last_fullsized":"2024-03-26T21:30:31.730593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:40.185136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:29:40.185136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":141,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.105552+0000","last_change":"2024-03-26T21:30:31.741495+0000","last_active":"2024-03-26T21:32:35.105552+0000","last_peered":"2024-03-26T21:32:35.105552+0000","last_clean":"2024-03-26T21:30:19.906082+0000","last_became_active":"2024-03-26T21:30:31.741495+0000","last_became_peered":"2024-03-26T21:30:31.741495+0000","last_unstale":"2024-03-26T21:32:35.105552+0000","last_undegraded":"2024-03-26T21:32:35.105552+0000","last_fullsized":"2024-03-26T21:30:31.732617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:39.867799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:29:39.867799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":120,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.137097+0000","last_change":"2024-03-26T21:30:31.741186+0000","last_active":"2024-03-26T21:32:35.137097+0000","last_peered":"2024-03-26T21:32:35.137097+0000","last_clean":"2024-03-26T21:28:42.246719+0000","last_became_active":"2024-03-26T21:30:31.741186+0000","last_became_peered":"2024-03-26T21:30:31.741186+0000","last_unstale":"2024-03-26T21:32:35.137097+0000","last_undegraded":"2024-03-26T21:30:31.735163+0000","last_fullsized":"2024-03-26T21:30:31.735041+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:30:19.267430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:19.267430+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"126'8221","reported_seq":8375,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:36.498040+0000","last_change":"2024-03-26T21:32:23.622753+0000","last_active":"2024-03-26T21:32:36.498040+0000","last_peered":"2024-03-26T21:32:36.498040+0000","last_clean":"2024-03-26T21:32:36.498040+0000","last_became_active":"2024-03-26T21:30:38.550667+0000","last_became_peered":"2024-03-26T21:30:38.550667+0000","last_unstale":"2024-03-26T21:32:36.498040+0000","last_undegraded":"2024-03-26T21:32:36.498040+0000","last_fullsized":"2024-03-26T21:32:36.498040+0000","mapping_epoch":95,"log_start":"105'2729","ondisk_log_start":"105'2729","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7423","last_scrub_stamp":"2024-03-26T21:32:23.622299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.622299+0000","objects_scrubbed":7320,"log_size":5492,"log_dups_size":0,"ondisk_log_size":5492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:33:39.296543+0000","scrub_duration":3638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434962432,"num_objects":6637,"num_object_clones":0,"num_object_copies":19911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8221,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":73,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.121584+0000","last_change":"2024-03-26T21:30:31.739518+0000","last_active":"2024-03-26T21:32:35.121584+0000","last_peered":"2024-03-26T21:32:35.121584+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:30:31.739518+0000","last_became_peered":"2024-03-26T21:30:31.739518+0000","last_unstale":"2024-03-26T21:32:35.121584+0000","last_undegraded":"2024-03-26T21:32:35.121584+0000","last_fullsized":"2024-03-26T21:30:31.730680+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:35.850889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":195,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-26T21:32:35.128036+0000","last_change":"2024-03-26T21:31:29.192047+0000","last_active":"2024-03-26T21:32:35.128036+0000","last_peered":"2024-03-26T21:32:35.128036+0000","last_clean":"2024-03-26T21:32:35.128036+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:32:35.128036+0000","last_undegraded":"2024-03-26T21:32:35.128036+0000","last_fullsized":"2024-03-26T21:32:35.128036+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:31:29.191678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.191678+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:32:42.551761+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":133,"reported_epoch":126,"state":"active+undersized","last_fresh":"2024-03-26T21:32:35.116646+0000","last_change":"2024-03-26T21:30:31.743458+0000","last_active":"2024-03-26T21:32:35.116646+0000","last_peered":"2024-03-26T21:32:35.116646+0000","last_clean":"2024-03-26T21:30:19.791720+0000","last_became_active":"2024-03-26T21:30:31.743458+0000","last_became_peered":"2024-03-26T21:30:31.743458+0000","last_unstale":"2024-03-26T21:32:35.116646+0000","last_undegraded":"2024-03-26T21:32:35.116646+0000","last_fullsized":"2024-03-26T21:30:31.731249+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:28:33.816134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":138,"reported_epoch":126,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:32:35.119601+0000","last_change":"2024-03-26T21:30:31.741371+0000","last_active":"2024-03-26T21:32:35.119601+0000","last_peered":"2024-03-26T21:32:35.119601+0000","last_clean":"2024-03-26T21:29:37.643762+0000","last_became_active":"2024-03-26T21:30:31.741371+0000","last_became_peered":"2024-03-26T21:30:31.741371+0000","last_unstale":"2024-03-26T21:32:35.119601+0000","last_undegraded":"2024-03-26T21:30:31.731807+0000","last_fullsized":"2024-03-26T21:30:31.731688+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:29:37.647105+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6970081304,"num_objects":106356,"num_object_clones":0,"num_object_copies":319068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106356,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132614,"num_write_kb":7646977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18725285888,"data_stored":18725273672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89333,"ondisk_log_size":89333,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":430,"ondisk_log_size":430,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070517,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2206028,"kb_used_data":2138752,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91539124,"statfs":{"total":95995035648,"available":93736062976,"internally_reserved":0,"allocated":2190082048,"data_stored":2189555239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199600,"kb_used_data":2136484,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91545552,"statfs":{"total":95995035648,"available":93742645248,"internally_reserved":0,"allocated":2187759616,"data_stored":2187259091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2678304,"kb_used_data":2566548,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91066848,"statfs":{"total":95995035648,"available":93252452352,"internally_reserved":0,"allocated":2628145152,"data_stored":2627596794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3544528,"kb_used_data":3427204,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":90200624,"statfs":{"total":95995035648,"available":92365438976,"internally_reserved":0,"allocated":3509456896,"data_stored":3508925321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2677796,"kb_used_data":2570200,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91067356,"statfs":{"total":95995035648,"available":93252972544,"internally_reserved":0,"allocated":2631884800,"data_stored":2631396110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3981644,"kb_used_data":3833860,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":89763508,"statfs":{"total":95995035648,"available":91917832192,"internally_reserved":0,"allocated":3925872640,"data_stored":3925339683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27632,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717520,"statfs":{"total":95995035648,"available":95966740480,"internally_reserved":0,"allocated":630784,"data_stored":285906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659198,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1782012,"kb_used_data":1718896,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91963140,"statfs":{"total":95995035648,"available":94170255360,"internally_reserved":0,"allocated":1760149504,"data_stored":1759636807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3488157696,"data_stored":3488153624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2613710848,"data_stored":2613706776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2610298880,"data_stored":2610298880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3915448320,"data_stored":3915448320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1747058688,"data_stored":1747058688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2174095360,"data_stored":2174091288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2176516096,"data_stored":2176516096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:32:40.078 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-26T21:32:40.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:40.155 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:40.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:40.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:40 smithi055 ceph-mon[40587]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:40 smithi055 ceph-mon[40587]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:40 smithi055 ceph-mon[29406]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:40 smithi055 ceph-mon[29406]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:40.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:40.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:40 smithi067 ceph-mon[33346]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:32:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:40 smithi067 ceph-mon[33346]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 33399/319662 objects degraded (10.448%) 2024-03-26T21:32:40.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:40.558 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:40.610 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:40.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:40.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:40.752 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:40.796 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:40.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:40.836 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:40.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:40.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:40.876 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:40.920 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:40.965 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:40.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:41.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:41.010 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:41.010 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:41.010 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:41.011 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:41.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:41.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:41.278 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:41.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3221725668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:32:41.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:32:41.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:32:41.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:32:41.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3221725668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:41 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:32:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:41 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:32:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:41 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:32:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:41 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:32:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3221725668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:41.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:41.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:41.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:41.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:41.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.847 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:41.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:41.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:41.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:42.021 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:32:42.021 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:32:42.021 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:32:42.022 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:32:42.022 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:32:42.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:42.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:32:42.140 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:32:42.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:42.237 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:42.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:42 smithi055 ceph-mon[40587]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 33399/315093 objects degraded (10.600%) 2024-03-26T21:32:42.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 33399/319662 objects degraded (10.448%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:42.309 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:42 smithi055 ceph-mon[29406]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 33399/315093 objects degraded (10.600%) 2024-03-26T21:32:42.309 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 33399/319662 objects degraded (10.448%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:42.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:42.378 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:42.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:42 smithi067 ceph-mon[33346]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 27 active+undersized+degraded, 35 active+undersized, 85 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s; 33399/315093 objects degraded (10.600%) 2024-03-26T21:32:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 33399/319662 objects degraded (10.448%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:42.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:42.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:42.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:42.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:32:42.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:42.947 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:42.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:43.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.101 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:43.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:43.164 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:43.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.250 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:43.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:43.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:43.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:43 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:32:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:43 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:32:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:43 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:32:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:43 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:32:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:43.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:32:43.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:32:43.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:32:43.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:32:43.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:32:43.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:32:43.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:32:43.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:43 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:32:43.610 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:43.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:43.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.789 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:43.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:43.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:43.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:43.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:44.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:44.073 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:44.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:32:44.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:44.222 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:32:44.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:44.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:44.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:44 smithi067 ceph-mon[33346]: pgmap v152: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3759349278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:44 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.524 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:44.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[40587]: pgmap v152: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:44.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3759349278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[29406]: pgmap v152: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3759349278' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:44 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:32:44.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.672 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:44.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.815 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:44.916 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:44.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:44.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:45.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:45.078 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:45.078 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-03-26T21:32:45.078 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-26T21:32:45.078 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-03-26T21:32:45.078 DEBUG:teuthology.orchestra.run.smithi067:osd.4> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T21:32:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:45.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl start ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4 2024-03-26T21:32:45.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:45.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:45.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:45.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:45.368 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:45.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:45.480 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:45 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:32:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:45 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:32:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:45 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:32:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:45 smithi067 ceph-mon[33346]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T21:32:45.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:45 smithi067 systemd[1]: Starting Ceph osd.4 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:32:45.504 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[29406]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:32:45.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:45 smithi055 ceph-mon[40587]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T21:32:45.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:45.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:45.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:45.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:45.844 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:45.884 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:45.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:45.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:46.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:46.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:46.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:46.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:46.338 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:46 smithi067 ceph-mon[33346]: pgmap v154: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 696 op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:46.338 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:46.338 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:46 smithi067 podman[76679]: 2024-03-26 21:32:46.096838682 +0000 UTC m=+0.027105321 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:32:46.338 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:46 smithi067 podman[76679]: 2024-03-26 21:32:46.215218214 +0000 UTC m=+0.145484837 container create ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:32:46.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:46.436 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:46.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:46.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:46 smithi055 ceph-mon[29406]: pgmap v154: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 696 op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:46.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:46.549 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:46 smithi055 ceph-mon[40587]: pgmap v154: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 6.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 696 op/s; 31866/304917 objects degraded (10.451%) 2024-03-26T21:32:46.549 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:32:46.567 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:46.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:46.698 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:46.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:46.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:46.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:46.898 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:46 smithi067 podman[76679]: 2024-03-26 21:32:46.697093326 +0000 UTC m=+0.627359959 container init ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-26T21:32:46.899 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:46 smithi067 podman[76679]: 2024-03-26 21:32:46.70289691 +0000 UTC m=+0.633163535 container start ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T21:32:46.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:46.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:32:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:32:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:32:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:32:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:927 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94208173348352 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:28:58.662586+0000", 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 228.317992285, 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0248679110000001, 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 927 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662586+0000", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662586+0000", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662588+0000", 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.694e-06 2024-03-26T21:32:47.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662600+0000", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2024e-05 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662618+0000", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7762e-05 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.662628+0000", 2024-03-26T21:32:47.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7410000000000004e-06 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.669830+0000", 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072024469999999998 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:58.669839+0000", 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4259999999999992e-06 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.678814+0000", 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0089749960000001 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.678980+0000", 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016565500000000001 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.680050+0000", 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001070178 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.680176+0000", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125709 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.680499+0000", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032274300000000001 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.680580+0000", 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1118999999999994e-05 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:28:59.687454+0000", 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068744169999999999 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1122 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94208173344896 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.733118+0000", 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 136.24746053000001, 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.008425468, 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1122 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733118+0000", 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733118+0000", 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733120+0000", 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.742e-06 2024-03-26T21:32:47.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733126+0000", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2389999999999999e-06 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733138+0000", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1374e-05 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733146+0000", 2024-03-26T21:32:47.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8589999999999994e-06 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733204+0000", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8094999999999999e-05 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.733215+0000", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0264e-05 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738971+0000", 2024-03-26T21:32:47.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0057566600000001 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739131+0000", 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015922899999999999 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.741544+0000", 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024130060000000001 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:9033 7.0 7:08c23c75:::benchmark_data_smithi055_81491_object9032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:56.665957+0000", 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 110.31462150599999, 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070539619999999997, 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 9033 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.665957+0000", 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.665957+0000", 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.665961+0000", 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8160000000000004e-06 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666056+0000", 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4822999999999997e-05 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666066+0000", 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6909999999999992e-06 2024-03-26T21:32:47.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666073+0000", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5039999999999996e-06 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666120+0000", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7321000000000003e-05 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666243+0000", 2024-03-26T21:32:47.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122516 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666364+0000", 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121641 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666933+0000", 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000568466 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.668380+0000", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001447219 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736379+0000", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067998477000000002 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736448+0000", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000002e-05 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736497+0000", 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8476000000000001e-05 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:9035 7.0 7:0aa0c845:::benchmark_data_smithi055_81491_object9034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:56.666499+0000", 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 110.314079455, 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070219499000000005, 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 9035 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666499+0000", 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666499+0000", 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666500+0000", 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2600000000000002e-07 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666594+0000", 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4450000000000005e-05 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666613+0000", 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8555000000000001e-05 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666623+0000", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0203e-05 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.666953+0000", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032994099999999999 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.667044+0000", 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.667145+0000", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101037 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.667655+0000", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051007400000000001 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.668832+0000", 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001177183 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736626+0000", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067794033000000004 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736669+0000", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2746e-05 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:56.736719+0000", 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9672999999999997e-05 2024-03-26T21:32:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51364 7.0 7:0df4495a:::benchmark_data_smithi055_81491_object51363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:20.556195+0000", 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.424383637000005, 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067996018000000005, 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.036 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51364 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556195+0000", 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556195+0000", 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556196+0000", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2610000000000001e-06 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556247+0000", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0624000000000002e-05 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556258+0000", 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1045e-05 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556264+0000", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9560000000000002e-06 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556312+0000", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8446000000000003e-05 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556455+0000", 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014310000000000001 2024-03-26T21:32:47.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.556554+0000", 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8548999999999994e-05 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.557925+0000", 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013707439999999999 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.558492+0000", 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056698199999999995 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.624074+0000", 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.065582352999999996 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.624142+0000", 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8155000000000001e-05 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.624191+0000", 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8803e-05 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51939 7.0 7:08bf17a4:::benchmark_data_smithi055_81491_object51938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:21.010836+0000", 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.969742420000003, 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048789322000000003, 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51939 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010836+0000", 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010836+0000", 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010837+0000", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.11e-06 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010899+0000", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2034999999999998e-05 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010909+0000", 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7890000000000006e-06 2024-03-26T21:32:47.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010916+0000", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.64e-06 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.010959+0000", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2994000000000002e-05 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.011061+0000", 2024-03-26T21:32:47.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102512 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.011217+0000", 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015583099999999999 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.013092+0000", 2024-03-26T21:32:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001874791 2024-03-26T21:32:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.013138+0000", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5728e-05 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.059549+0000", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046411422000000001 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.059589+0000", 2024-03-26T21:32:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0009999999999998e-05 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.059626+0000", 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.646e-05 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51968 7.0 7:0bd2e225:::benchmark_data_smithi055_81491_object51967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:21.071549+0000", 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.909029548000007, 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037261820000000001, 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51968 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071549+0000", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071549+0000", 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071551+0000", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169999999999999e-06 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071594+0000", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3733000000000003e-05 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071605+0000", 2024-03-26T21:32:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0574000000000001e-05 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071613+0000", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6930000000000007e-06 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071657+0000", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4237999999999999e-05 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071777+0000", 2024-03-26T21:32:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011965599999999999 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.071920+0000", 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014337599999999999 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.074338+0000", 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002418173 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.074714+0000", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037557900000000002 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.108697+0000", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033982908999999999 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.108760+0000", 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3646999999999996e-05 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.108811+0000", 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0624999999999997e-05 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51971 7.0 7:0754066c:::benchmark_data_smithi055_81491_object51970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:21.072098+0000", 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.908480479999994, 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040684931000000001, 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51971 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072098+0000", 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072098+0000", 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072099+0000", 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2699999999999998e-07 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072152+0000", 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3016000000000002e-05 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072163+0000", 2024-03-26T21:32:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1283e-05 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.072172+0000", 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4270000000000008e-06 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.073113+0000", 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00094102699999999999 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.073240+0000", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127601 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.073333+0000", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2534000000000005e-05 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.074439+0000", 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001105591 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.074870+0000", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043107699999999998 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.112679+0000", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037809725000000002 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.112737+0000", 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7624999999999998e-05 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:21.112783+0000", 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6198000000000001e-05 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:62097 7.0 7:01f56305:::benchmark_data_smithi055_81491_object62096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:28.292982+0000", 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.687596915, 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.063104777000000001, 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62097 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.292982+0000", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.292982+0000", 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.292984+0000", 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8980000000000001e-06 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293035+0000", 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1387000000000001e-05 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293046+0000", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1284000000000001e-05 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293054+0000", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0690000000000002e-06 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293102+0000", 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7413999999999997e-05 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293226+0000", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124571 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.293340+0000", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011319600000000001 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.295275+0000", 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001935719 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.296983+0000", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00170755 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.356001+0000", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.059018503999999999 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.356054+0000", 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2206000000000001e-05 2024-03-26T21:32:47.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:28.356087+0000", 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2978999999999997e-05 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:84401 7.0 7:02f11fee:::benchmark_data_smithi055_81491_object84400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:32:47.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:47.514983+0000", 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.465595182999998, 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056036146000000002, 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84401 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.514983+0000", 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.514983+0000", 2024-03-26T21:32:47.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.514985+0000", 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0920000000000001e-06 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515195+0000", 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021045499999999999 2024-03-26T21:32:47.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515207+0000", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-05 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515216+0000", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2919999999999996e-06 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515269+0000", 2024-03-26T21:32:47.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3851999999999999e-05 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515416+0000", 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146458 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.515557+0000", 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014113299999999999 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.518269+0000", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027123889999999999 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.520389+0000", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021199959999999999 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.570938+0000", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050548485999999997 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.570986+0000", 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7618999999999999e-05 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.571020+0000", 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4073999999999999e-05 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:84407 7.0 7:03a27a80:::benchmark_data_smithi055_81491_object84406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:47.521241+0000", 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.459337947999998, 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049873219000000003, 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84407 2024-03-26T21:32:47.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521241+0000", 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521241+0000", 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521242+0000", 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5600000000000001e-06 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521290+0000", 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8199999999999999e-05 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521301+0000", 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0203e-05 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521307+0000", 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200000000000001e-06 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521345+0000", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8266999999999998e-05 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521456+0000", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111075 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.521557+0000", 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010117 2024-03-26T21:32:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.523559+0000", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002001685 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.523991+0000", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000431836 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.571059+0000", 2024-03-26T21:32:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047068098000000003 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.571091+0000", 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2146e-05 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:47.571114+0000", 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3059000000000001e-05 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:90711 7.0 7:01b44481:::benchmark_data_smithi055_81491_object90710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:53.369621+0000", 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.610957898999999, 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035476187999999999, 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 90711 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369621+0000", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369621+0000", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369622+0000", 2024-03-26T21:32:47.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4729999999999999e-06 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369668+0000", 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5272999999999998e-05 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369681+0000", 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3458999999999999e-05 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369689+0000", 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6059999999999997e-06 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369738+0000", 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9502999999999997e-05 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369835+0000", 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7003999999999995e-05 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.369936+0000", 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010071199999999999 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.374133+0000", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041969099999999999 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.386843+0000", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012710335 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.405022+0000", 2024-03-26T21:32:47.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018179007 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.405065+0000", 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2996e-05 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:53.405097+0000", 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1909999999999998e-05 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:98832 7.0 7:0c5a726b:::benchmark_data_smithi055_81491_object98831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:01.082945+0000", 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897633476999999, 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041163007000000001, 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 98832 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.082945+0000", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.082945+0000", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.082947+0000", 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.33e-06 2024-03-26T21:32:47.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.082996+0000", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9288e-05 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083006+0000", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0482999999999999e-05 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083012+0000", 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9649999999999996e-06 2024-03-26T21:32:47.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083059+0000", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6523999999999997e-05 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083154+0000", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5218999999999998e-05 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083256+0000", 2024-03-26T21:32:47.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101683 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.089269+0000", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060128070000000002 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.093814+0000", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004545739 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.123987+0000", 2024-03-26T21:32:47.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030173113000000001 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124055+0000", 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7997999999999995e-05 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124108+0000", 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2858000000000001e-05 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:98834 7.0 7:04ca0e92:::benchmark_data_smithi055_81491_object98833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:01.083030+0000", 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897548395999998, 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041241488999999999, 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 98834 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083030+0000", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083030+0000", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083031+0000", 2024-03-26T21:32:47.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1800000000000005e-07 2024-03-26T21:32:47.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083060+0000", 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9328000000000001e-05 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083067+0000", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0740000000000004e-06 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.083071+0000", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0579999999999999e-06 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.085488+0000", 2024-03-26T21:32:47.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024162089999999999 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.085590+0000", 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010222599999999999 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.085688+0000", 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8022000000000004e-05 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.089275+0000", 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003586671 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.093854+0000", 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00457954 2024-03-26T21:32:47.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124168+0000", 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030314260999999999 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124227+0000", 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8189000000000001e-05 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124272+0000", 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5192999999999999e-05 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:98837 7.0 7:0b372f7d:::benchmark_data_smithi055_81491_object98836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:01.084268+0000", 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.896310595000003, 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040111417000000003, 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 98837 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084268+0000", 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084268+0000", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084269+0000", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8899999999999999e-07 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084330+0000", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1686999999999994e-05 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084352+0000", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1370999999999999e-05 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.084363+0000", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0910000000000001e-05 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.087583+0000", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032206140000000001 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.087666+0000", 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2714999999999997e-05 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.087775+0000", 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109239 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.093829+0000", 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006053541 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.108385+0000", 2024-03-26T21:32:47.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014556052 2024-03-26T21:32:47.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124321+0000", 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015935964 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124355+0000", 2024-03-26T21:32:47.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3710000000000001e-05 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.124380+0000", 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4924999999999999e-05 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:98848 7.0 7:0c69de51:::benchmark_data_smithi055_81491_object98847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:01.095331+0000", 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.885247616000001, 2024-03-26T21:32:47.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039062247000000001, 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 98848 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095331+0000", 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095331+0000", 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095332+0000", 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.246e-06 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095377+0000", 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4860000000000001e-05 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095387+0000", 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6779999999999995e-06 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095394+0000", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6279999999999996e-06 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095441+0000", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6539000000000003e-05 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095524+0000", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3159999999999997e-05 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.095644+0000", 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120216 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.098086+0000", 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024419569999999998 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.108494+0000", 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407415 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.134123+0000", 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025629543000000001 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.134326+0000", 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020296299999999999 2024-03-26T21:32:47.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:01.134393+0000", 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7041999999999999e-05 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118946 7.0 7:0cdf0469:::benchmark_data_smithi055_81491_object118945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.112150+0000", 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.868429038999999, 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068417741000000004, 2024-03-26T21:32:47.123 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118946 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112150+0000", 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112150+0000", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112151+0000", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1500000000000003e-07 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112200+0000", 2024-03-26T21:32:47.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9648999999999998e-05 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112213+0000", 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2646999999999999e-05 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112221+0000", 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5720000000000003e-06 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112270+0000", 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8310999999999999e-05 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112416+0000", 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146687 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112588+0000", 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171662 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.114889+0000", 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023007829999999998 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116088+0000", 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001198693 2024-03-26T21:32:47.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.180480+0000", 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.064392668 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.180519+0000", 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8603000000000003e-05 2024-03-26T21:32:47.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.180567+0000", 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8551000000000003e-05 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:119232 7.0 7:09eaa236:::benchmark_data_smithi055_81491_object119231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.441365+0000", 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.539213582999999, 2024-03-26T21:32:47.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038187196999999999, 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 119232 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441365+0000", 2024-03-26T21:32:47.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441365+0000", 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441366+0000", 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441414+0000", 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7553999999999999e-05 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441422+0000", 2024-03-26T21:32:47.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441427+0000", 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3410000000000002e-06 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441488+0000", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0483999999999999e-05 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441606+0000", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011843199999999999 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.441730+0000", 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012339299999999999 2024-03-26T21:32:47.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.443911+0000", 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021814059999999999 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.444243+0000", 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033218599999999999 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.479483+0000", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035239274000000001 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.479521+0000", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8105999999999997e-05 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.479552+0000", 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1674000000000002e-05 2024-03-26T21:32:47.137 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:120747 7.0 7:0c0d8ed6:::benchmark_data_smithi055_81491_object1260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:24.848340+0000", 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.132238189999999, 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040606945999999998, 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.138 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 120747 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848340+0000", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848340+0000", 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848341+0000", 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2200000000000004e-07 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848344+0000", 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5900000000000002e-06 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848353+0000", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1290000000000002e-06 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.848359+0000", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.28e-06 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.851249+0000", 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028896289999999999 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.851381+0000", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000132566 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.851519+0000", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013723500000000001 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.853679+0000", 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002160253 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.856288+0000", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026090190000000002 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.888823+0000", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032535499000000002 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.888886+0000", 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2215e-05 2024-03-26T21:32:47.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:24.888947+0000", 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1909000000000006e-05 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:120930 7.0 7:06942a65:::benchmark_data_smithi055_81491_object1443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:25.073507+0000", 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.907071845000001, 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049871388000000003, 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:32:47.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 120930 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073507+0000", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:32:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073507+0000", 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073509+0000", 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8279999999999999e-06 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073515+0000", 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2339999999999999e-06 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073529+0000", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.366e-05 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073538+0000", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0629999999999995e-06 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073586+0000", 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8693999999999998e-05 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073666+0000", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9208000000000005e-05 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.073750+0000", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4083000000000001e-05 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.075871+0000", 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021211350000000001 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.077069+0000", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011986270000000001 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.123273+0000", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046203408000000001 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.123334+0000", 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0857999999999999e-05 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:32:47.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:25.123378+0000", 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4589999999999998e-05 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:32:47.150 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:32:47.150 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:46 smithi067 podman[76679]: 2024-03-26 21:32:46.905965465 +0000 UTC m=+0.836232111 container attach ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T21:32:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:47.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:47.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:47.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:47.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:32:47.385 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:47.445 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 31866/304917 objects degraded (10.451%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-mon[33346]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T21:32:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:47.489 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate[76724]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:32:47.490 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 bash[76679]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T21:32:47.491 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:47 smithi067 podman[76679]: 2024-03-26 21:32:47.382495548 +0000 UTC m=+1.312762181 container died ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:32:47.549 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:47.557 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:47.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 31866/304917 objects degraded (10.451%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:47.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[29406]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T21:32:47.557 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:47.558 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 31866/304917 objects degraded (10.451%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:47.558 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[40587]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T21:32:47.558 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:47.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.606 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:47.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:47.656 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:47.701 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:47.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:47.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:47.740 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:47.744 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:47.782 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:47.826 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:47.831 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:47.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:47.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:47.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:47.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:48.025 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:48.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:48.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:48.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:48.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:48.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:48.489 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:48 smithi067 podman[76937]: 2024-03-26 21:32:48.367167379 +0000 UTC m=+0.974267831 container remove ddfbdc83003840d31d6bd2332e4530c1dc05ae9ec8fcd62562cee97df02e4712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-activate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T21:32:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: pgmap v156: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2941468328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:48 smithi067 ceph-mon[33346]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T21:32:48.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:48.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: pgmap v156: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2941468328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[29406]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: pgmap v156: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2941468328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-mon[40587]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T21:32:48.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:48.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:32:48.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:32:48.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:48.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:48.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:48.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:48.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:49.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:32:49.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:49.115 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:49.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:49.239 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 podman[77099]: 2024-03-26 21:32:49.046392969 +0000 UTC m=+0.027055051 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:32:49.239 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 podman[77099]: 2024-03-26 21:32:49.155335492 +0000 UTC m=+0.135997580 container create 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:32:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:49.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:49.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:49.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:49.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.535 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:49.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:49.543 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 podman[77099]: 2024-03-26 21:32:49.433858952 +0000 UTC m=+0.414521056 container init 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True) 2024-03-26T21:32:49.543 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 podman[77099]: 2024-03-26 21:32:49.438429312 +0000 UTC m=+0.419091396 container start 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:32:49.544 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 ceph-osd[77177]: -- 172.21.15.67:0/1182505387 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55682d8fb680 con 0x55682d8c5000 2024-03-26T21:32:49.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:32:49.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:49.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:49.679 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:49.773 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:49.810 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 bash[77099]: 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 2024-03-26T21:32:49.810 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 systemd[1]: Started Ceph osd.4 for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:32:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.811 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:49.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:49.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:49.942 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:49.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:50.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:50.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:50.097 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:49 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:32:49.980+0000 7f87200587c0 -1 Falling back to public interface 2024-03-26T21:32:50.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.199 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:50.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:50.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:50.429 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:50.465 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:50.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:50.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:50.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:50 smithi067 ceph-mon[33346]: pgmap v158: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 895 op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:50.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:32:50.789 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:50 smithi055 ceph-mon[29406]: pgmap v158: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 895 op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:50.790 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:50 smithi055 ceph-mon[40587]: pgmap v158: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 895 op/s; 30175/288804 objects degraded (10.448%) 2024-03-26T21:32:50.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:50.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:50.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:51.016 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:51.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:51.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:51.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:51.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.393 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:51.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:51.522 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:51.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.654 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:51.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:51.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:51.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:51.910 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:32:51.910 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:32:51.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:51.918 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:51.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:52.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:52.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:52.179 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:52.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:52.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:52.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:52.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:52.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:52.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:52 smithi067 ceph-mon[33346]: pgmap v159: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 980 op/s; 30175/283980 objects degraded (10.626%) 2024-03-26T21:32:52.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 30175/283980 objects degraded (10.626%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:52.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:52 smithi067 ceph-mon[33346]: osdmap e131: 8 total, 7 up, 8 in 2024-03-26T21:32:52.705 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:52.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:52.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[29406]: pgmap v159: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 980 op/s; 30175/283980 objects degraded (10.626%) 2024-03-26T21:32:52.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 30175/283980 objects degraded (10.626%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:52.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[29406]: osdmap e131: 8 total, 7 up, 8 in 2024-03-26T21:32:52.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[40587]: pgmap v159: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 980 op/s; 30175/283980 objects degraded (10.626%) 2024-03-26T21:32:52.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 30175/283980 objects degraded (10.626%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:52.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:52 smithi055 ceph-mon[40587]: osdmap e131: 8 total, 7 up, 8 in 2024-03-26T21:32:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.846 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:52.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:52.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:52.954 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:52 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:32:52.682+0000 7f87200587c0 -1 osd.4 92 log_to_monitors true 2024-03-26T21:32:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:52.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:53.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:53.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:53.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:53.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.393 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:53.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:53.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:53.615 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:53 smithi067 ceph-mon[33346]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.616 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:53 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.616 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:32:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:32:53.513+0000 7f87114b0700 -1 osd.4 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:32:53.627 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:53.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:32:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.674 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:53.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:53.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:53.780 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:53 smithi055 ceph-mon[29406]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.781 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:53 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.781 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:53 smithi055 ceph-mon[40587]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.781 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:53 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T21:32:53.824 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:53.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:53.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:53.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:54.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:54.100 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:54.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:54.224 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:54.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:54.298 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:32:54.298 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:32:54.298 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-26T21:32:54.298 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:32:54.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.372 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:54.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:54.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:54.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:54.614 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:54.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[40587]: pgmap v161: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 754 B/s rd, 0 B/s wr, 970 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:54.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[40587]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[40587]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[40587]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[29406]: pgmap v161: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 754 B/s rd, 0 B/s wr, 970 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[29406]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[29406]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T21:32:54.620 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:54 smithi055 ceph-mon[29406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.641 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:54 smithi067 ceph-mon[33346]: pgmap v161: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 754 B/s rd, 0 B/s wr, 970 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:54 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T21:32:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:54 smithi067 ceph-mon[33346]: from='osd.4 v2:172.21.15.67:6800/254752278' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:54 smithi067 ceph-mon[33346]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T21:32:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:54 smithi067 ceph-mon[33346]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:32:54.743 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:54.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:54.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:54.892 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:54.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:54.993 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:55.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:55.105 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:55.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:55.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T21:32:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:55.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:55.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:55.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:55.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:55.402 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:55.412 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:55.459 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:32:55.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.529 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:55.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:55.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:55.654 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: osd.4 v2:172.21.15.67:6800/254752278 boot 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4196374056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:55.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:55 smithi067 ceph-mon[33346]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:32:55.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:55.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:55.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:32:55.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: osd.4 v2:172.21.15.67:6800/254752278 boot 2024-03-26T21:32:55.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4196374056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[40587]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: osd.4 v2:172.21.15.67:6800/254752278 boot 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T21:32:55.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4196374056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:32:55.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:32:55.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:32:55.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:55 smithi055 ceph-mon[29406]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T21:32:55.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:55.906 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:55.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:32:56.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:56.032 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:56.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:56.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:56.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:56.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:32:56.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:32:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.411 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:56.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:32:56.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.537 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:56.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:56.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:56.663 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:32:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:32:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-mon[33346]: pgmap v164: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3372686239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-mon[33346]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:32:56.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:32:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:32:56.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:32:56.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:32:56.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[29406]: pgmap v164: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:56.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3372686239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:56.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[29406]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:32:56.741 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:32:56.741 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:32:56.741 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[40587]: pgmap v164: 148 pgs: 27 active+undersized+degraded, 35 active+undersized, 86 active+clean; 5.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 28513/273099 objects degraded (10.441%) 2024-03-26T21:32:56.741 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3372686239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:32:56.741 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:56 smithi055 ceph-mon[40587]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T21:32:56.741 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:32:56.741 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:32:56.741 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:32:56.741 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:32:56.742 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:32:56.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:56.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:56.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:56.940 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:56.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:57.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:57.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:57.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:57.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:57.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:57.382 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:32:57.382 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:32:57.382 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:32:57.382 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:32:57.382 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:32:57.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:57.507 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:57.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:57.618 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9334 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9451 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9610 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9697 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9487 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 28513/273099 objects degraded (10.441%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9334 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9451 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9610 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9697 2024-03-26T21:32:57.619 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9487 2024-03-26T21:32:57.620 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 28513/273099 objects degraded (10.441%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:57.620 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:57.640 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9334 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9451 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9610 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9697 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 132'9487 2024-03-26T21:32:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 28513/273099 objects degraded (10.441%), 27 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:32:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:32:57.742 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:57.786 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:57.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:32:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:57.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:32:58.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:58.052 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:58.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:32:58.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:32:58.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:32:58.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:58.337 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:58.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:58.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:32:58.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:32:58 smithi055 ceph-mon[29406]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfilling, 12 active+undersized, 121 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 27483/263397 objects degraded (10.434%) 2024-03-26T21:32:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:32:58 smithi055 ceph-mon[40587]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfilling, 12 active+undersized, 121 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 27483/263397 objects degraded (10.434%) 2024-03-26T21:32:58.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:58.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:32:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:58.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:58.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:32:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:32:58 smithi067 ceph-mon[33346]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfilling, 12 active+undersized, 121 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 27483/263397 objects degraded (10.434%) 2024-03-26T21:32:58.750 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:32:58.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:58.899 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:58.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:59.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:32:59.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:32:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:32:59.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:59.303 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.303 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:32:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:32:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:32:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:59.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:32:59.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:32:59.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:32:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:32:59.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:32:59.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:59.732 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:59.783 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:32:59.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:32:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:32:59.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:32:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:32:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:32:59.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:00.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:00.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:00.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:00.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:00.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:00.502 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T21:33:00.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:00.531 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:33:00.531 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:33:00.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1123 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94208173341440 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.80313759699999, 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0115813600000001, 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:33:00.532 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1123 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732483+0000", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2589999999999999e-06 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732488+0000", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7539999999999997e-06 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732499+0000", 2024-03-26T21:33:00.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1243000000000001e-05 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732508+0000", 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0040000000000005e-06 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732567+0000", 2024-03-26T21:33:00.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8811000000000003e-05 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732573+0000", 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2330000000000003e-06 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736258+0000", 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036856459999999999 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736267+0000", 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7460000000000006e-06 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739066+0000", 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.002799072 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739206+0000", 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013932300000000001 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.744063+0000", 2024-03-26T21:33:00.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004857269 2024-03-26T21:33:00.539 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1124 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94208178013312 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.80297536399999, 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.011248098, 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:33:00.540 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1124 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:33:00.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:33:00.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:33:00.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4099999999999998e-07 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732648+0000", 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.771e-06 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732657+0000", 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9639999999999992e-06 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732679+0000", 2024-03-26T21:33:00.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1985999999999999e-05 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732893+0000", 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021387399999999999 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:00.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732903+0000", 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9499999999999996e-06 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738098+0000", 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005195499 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738271+0000", 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172795 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.743892+0000", 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005620618 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51352 7.7 7:e1b9e060:::benchmark_data_smithi055_81491_object51351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:33:00.549 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:20.548726+0000", 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 99.986893322, 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070866567000000005, 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51352 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548726+0000", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548726+0000", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548727+0000", 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7000000000000004e-07 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548789+0000", 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2539999999999994e-05 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548798+0000", 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2159999999999995e-06 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548805+0000", 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1389999999999998e-06 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548859+0000", 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3356999999999998e-05 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.548995+0000", 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136468 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.549114+0000", 2024-03-26T21:33:00.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011917399999999999 2024-03-26T21:33:00.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.550617+0000", 2024-03-26T21:33:00.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015025049999999999 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.553442+0000", 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028254589999999998 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.619506+0000", 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066063698000000004 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.619544+0000", 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7916000000000001e-05 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.619592+0000", 2024-03-26T21:33:00.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8325000000000002e-05 2024-03-26T21:33:00.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:51356 7.c 7:38c1e1f5:::benchmark_data_smithi055_81491_object51355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:31:20.552901+0000", 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 99.982717785999995, 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070450648000000005, 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 51356 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552901+0000", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552901+0000", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552902+0000", 2024-03-26T21:33:00.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4399999999999998e-07 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552955+0000", 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3028000000000001e-05 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552965+0000", 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8570000000000007e-06 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.552973+0000", 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6510000000000001e-06 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.553023+0000", 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9762999999999998e-05 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.553180+0000", 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015696199999999999 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.553338+0000", 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015836100000000001 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.555613+0000", 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022749889999999998 2024-03-26T21:33:00.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.555780+0000", 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016731 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.623276+0000", 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067495654000000002 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.623310+0000", 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4063000000000002e-05 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:31:20.623352+0000", 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2066e-05 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118015 7.c 7:3ece5431:::benchmark_data_smithi055_81491_object118014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.599139139000002, 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12752479, 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118015 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936489+0000", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6119999999999999e-06 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936522+0000", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3831999999999997e-05 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936531+0000", 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0749999999999998e-06 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936537+0000", 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7549999999999997e-06 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936603+0000", 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6093999999999994e-05 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936788+0000", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018466199999999999 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936962+0000", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017412300000000001 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942702+0000", 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057399269999999997 2024-03-26T21:33:00.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942741+0000", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9239999999999997e-05 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.063918+0000", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121176487 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.063965+0000", 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7635e-05 2024-03-26T21:33:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064005+0000", 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9347999999999997e-05 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118016 7.c 7:32014175:::benchmark_data_smithi055_81491_object118015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:33:00.573 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.599010456999999, 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12750198700000001, 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118016 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.68e-07 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936637+0000", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7277999999999999e-05 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936644+0000", 2024-03-26T21:33:00.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-06 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936651+0000", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8950000000000001e-06 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940183+0000", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035317270000000001 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940344+0000", 2024-03-26T21:33:00.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016089000000000001 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940539+0000", 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019523199999999999 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942824+0000", 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022846260000000001 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942871+0000", 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7315999999999999e-05 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064045+0000", 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121174358 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064082+0000", 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6288000000000002e-05 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064111+0000", 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9005000000000001e-05 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118146 7.7 7:e950c474:::benchmark_data_smithi055_81491_object118145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:22.193223+0000", 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.342396589000003, 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089940537000000001, 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118146 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193223+0000", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193223+0000", 2024-03-26T21:33:00.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193223+0000", 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5099999999999998e-07 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193284+0000", 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0715999999999999e-05 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193296+0000", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1756000000000001e-05 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193304+0000", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7519999999999996e-06 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193359+0000", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5622999999999999e-05 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193482+0000", 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012291799999999999 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.193637+0000", 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015440399999999999 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.196348+0000", 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027112429999999999 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.248545+0000", 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052197351000000003 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.283064+0000", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034519213 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.283127+0000", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2990999999999994e-05 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.283163+0000", 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5719000000000001e-05 2024-03-26T21:33:00.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118802 7.7 7:ec6db623:::benchmark_data_smithi055_81491_object118801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:22.942057+0000", 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.593562370999997, 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.087831959000000001, 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.588 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118802 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942057+0000", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942057+0000", 2024-03-26T21:33:00.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942058+0000", 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.319e-06 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942102+0000", 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3523000000000003e-05 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942114+0000", 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2035000000000001e-05 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942121+0000", 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7689999999999996e-06 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942186+0000", 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4290999999999998e-05 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942345+0000", 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015952300000000001 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.942501+0000", 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015603899999999999 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.945459+0000", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029578680000000002 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.957016+0000", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011556528 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.029822+0000", 2024-03-26T21:33:00.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072805849000000006 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.029857+0000", 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5107000000000001e-05 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.029889+0000", 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2107999999999998e-05 2024-03-26T21:33:00.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118932 7.7 7:ef1ef4f7:::benchmark_data_smithi055_81491_object118931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.098777+0000", 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.436842304000002, 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072757850999999998, 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118932 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098777+0000", 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098777+0000", 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098778+0000", 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7800000000000001e-07 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098829+0000", 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1829000000000001e-05 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098839+0000", 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5310000000000007e-06 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098844+0000", 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5790000000000003e-06 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.098896+0000", 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1767000000000001e-05 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.099033+0000", 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136618 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.099183+0000", 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150548 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.105022+0000", 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058385779999999997 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.171056+0000", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066034437000000001 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.171471+0000", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041504299999999998 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.171506+0000", 2024-03-26T21:33:00.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4465000000000003e-05 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.171535+0000", 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8778000000000001e-05 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118934 7.7 7:e3743f44:::benchmark_data_smithi055_81491_object118933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.602 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.101477+0000", 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.434141928999999, 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091138342999999997, 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118934 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101477+0000", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101477+0000", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101478+0000", 2024-03-26T21:33:00.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1999999999999998e-07 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101530+0000", 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1583999999999999e-05 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101539+0000", 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6849999999999996e-06 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.101545+0000", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7980000000000002e-06 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.102059+0000", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051390799999999999 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.102148+0000", 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8911000000000003e-05 2024-03-26T21:33:00.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.102246+0000", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8420999999999998e-05 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.108011+0000", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057644769999999996 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.185786+0000", 2024-03-26T21:33:00.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.077775222000000005 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.192544+0000", 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067579720000000001 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.192582+0000", 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8355999999999997e-05 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.192616+0000", 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3188999999999997e-05 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118941 7.7 7:eee613bb:::benchmark_data_smithi055_81491_object118940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.103749+0000", 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.431870496999998, 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.095251588999999998, 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.609 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118941 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103749+0000", 2024-03-26T21:33:00.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103749+0000", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103750+0000", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0389999999999999e-06 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103794+0000", 2024-03-26T21:33:00.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4406000000000001e-05 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103807+0000", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2768999999999999e-05 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.103815+0000", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6739999999999997e-06 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.105218+0000", 2024-03-26T21:33:00.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014037399999999999 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.105349+0000", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013101599999999999 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.105499+0000", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149934 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.108165+0000", 2024-03-26T21:33:00.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026656660000000001 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.186310+0000", 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.078145109000000004 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.198924+0000", 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012614372 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.198963+0000", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9048999999999998e-05 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199000+0000", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6814999999999999e-05 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118945 7.7 7:e6551a62:::benchmark_data_smithi055_81491_object118944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.112108+0000", 2024-03-26T21:33:00.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.423511157999997, 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.086998412999999997, 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118945 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112108+0000", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112108+0000", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112109+0000", 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1119999999999999e-06 2024-03-26T21:33:00.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112198+0000", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8442999999999993e-05 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112213+0000", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5294999999999998e-05 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112221+0000", 2024-03-26T21:33:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6300000000000004e-06 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112276+0000", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4418e-05 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112416+0000", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014055799999999999 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.112588+0000", 2024-03-26T21:33:00.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171671 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.115446+0000", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002857619 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.186350+0000", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070903965999999999 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199047+0000", 2024-03-26T21:33:00.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012697208 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199081+0000", 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4072000000000002e-05 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199106+0000", 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5420999999999999e-05 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118947 7.7 7:e2c5439f:::benchmark_data_smithi055_81491_object118946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:23.116220+0000", 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.419399536, 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.082976333999999999, 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118947 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116220+0000", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116220+0000", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116220+0000", 2024-03-26T21:33:00.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5199999999999996e-07 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116269+0000", 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8715999999999999e-05 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116279+0000", 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0423e-05 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116286+0000", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0650000000000004e-06 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116323+0000", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7011000000000002e-05 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116404+0000", 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1886999999999997e-05 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.116501+0000", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6821e-05 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.118053+0000", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015518559999999999 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.186412+0000", 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068359210000000004 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199135+0000", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012722687999999999 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199173+0000", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7840999999999999e-05 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:23.199196+0000", 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3064000000000001e-05 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:140426 7.7 7:e3443258:::benchmark_data_smithi055_81491_object20939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.175370+0000", 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.360249178, 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.082231068000000004, 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:00.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140426 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175370+0000", 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175370+0000", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175371+0000", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8000000000000005e-07 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175375+0000", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9210000000000002e-06 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175383+0000", 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3100000000000001e-06 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175389+0000", 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.852e-06 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175432+0000", 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3367e-05 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175548+0000", 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115942 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.175660+0000", 2024-03-26T21:33:00.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111737 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.180685+0000", 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050250989999999999 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.182694+0000", 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020095260000000002 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.257506+0000", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.074811242 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.257561+0000", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4829e-05 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.257601+0000", 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0463000000000002e-05 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:140432 7.7 7:e323a86c:::benchmark_data_smithi055_81491_object20945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.177185+0000", 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.358434083999999, 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.081260443000000002, 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.637 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140432 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177185+0000", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177185+0000", 2024-03-26T21:33:00.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177186+0000", 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3399999999999998e-07 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177189+0000", 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5700000000000001e-06 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177197+0000", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6229999999999997e-06 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177202+0000", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1599999999999997e-06 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.177922+0000", 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072012699999999999 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.178001+0000", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8720999999999994e-05 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.178113+0000", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111698 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.180790+0000", 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026773500000000002 2024-03-26T21:33:00.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.182804+0000", 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020138589999999998 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258349+0000", 2024-03-26T21:33:00.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075544815000000001 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258404+0000", 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5680999999999998e-05 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258445+0000", 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1105e-05 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:140437 7.7 7:e94ca87f:::benchmark_data_smithi055_81491_object20950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.184715+0000", 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.350904369999999, 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.08354926, 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140437 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184715+0000", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184715+0000", 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184716+0000", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.097e-06 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184721+0000", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3600000000000004e-06 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184732+0000", 2024-03-26T21:33:00.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1068e-05 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184738+0000", 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8100000000000003e-06 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184783+0000", 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5241000000000003e-05 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184862+0000", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8504999999999995e-05 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251323+0000", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066461377000000002 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258460+0000", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071368020000000002 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.264955+0000", 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064951039999999998 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.268160+0000", 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032044909999999998 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.268217+0000", 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7843999999999997e-05 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.268264+0000", 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6560999999999997e-05 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:140438 7.7 7:e48ff35c:::benchmark_data_smithi055_81491_object20951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.184749+0000", 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.350869809999999, 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.092567814999999998, 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140438 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184749+0000", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184749+0000", 2024-03-26T21:33:00.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184750+0000", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5999999999999999e-07 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184752+0000", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1689999999999999e-06 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184756+0000", 2024-03-26T21:33:00.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.184759+0000", 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3909999999999998e-06 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.254372+0000", 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069613123999999998 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.254492+0000", 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120327 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.254614+0000", 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012187 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258466+0000", 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038514119999999998 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.265041+0000", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065755930000000002 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.277219+0000", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012177698000000001 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.277273+0000", 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.435e-05 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.277317+0000", 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3816999999999997e-05 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:140436 7.1 e127/95 7:80dd6d69:::benchmark_data_smithi055_81491_object20949:head v 127'8652, mlcod=127'8649)", 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.185138+0000", 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.350480701999999, 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073252266999999996, 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.185138+0000", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.185138+0000", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.185139+0000", 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9699999999999996e-07 2024-03-26T21:33:00.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.185143+0000", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0720000000000001e-06 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251169+0000", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066026478999999999 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251184+0000", 2024-03-26T21:33:00.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4362000000000001e-05 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251236+0000", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2453000000000001e-05 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251264+0000", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7441999999999999e-05 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.257693+0000", 2024-03-26T21:33:00.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064293400000000004 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258391+0000", 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069762199999999996 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:140440 7.c 7:3dd0c4e1:::benchmark_data_smithi055_81491_object20953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.188228+0000", 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.347391121000001, 2024-03-26T21:33:00.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073843790000000006, 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140440 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.188228+0000", 2024-03-26T21:33:00.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.188228+0000", 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.188230+0000", 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7940000000000001e-06 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.188239+0000", 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6220000000000004e-06 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251174+0000", 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.062934688000000003 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251190+0000", 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5625e-05 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251450+0000", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026011200000000001 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251559+0000", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108775 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.251660+0000", 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100929 2024-03-26T21:33:00.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258583+0000", 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069237259999999998 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.258597+0000", 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3794e-05 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.261972+0000", 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033749180000000002 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.262033+0000", 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0897999999999999e-05 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.262072+0000", 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8909000000000003e-05 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:141122 7.d 7:b3097546:::benchmark_data_smithi055_81491_object21635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:44.881535+0000", 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.654084448999999, 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.07076839, 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:00.667 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141122 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881535+0000", 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881535+0000", 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:00.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881536+0000", 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6139999999999999e-06 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881542+0000", 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8019999999999997e-06 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881550+0000", 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3089999999999996e-06 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881557+0000", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3180000000000003e-06 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.881883+0000", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032603800000000001 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.882009+0000", 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012597899999999999 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,1", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.882135+0000", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012615200000000001 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.885155+0000", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030196899999999998 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.892149+0000", 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069948889999999998 2024-03-26T21:33:00.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.952216+0000", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060066630000000003 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.952266+0000", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9731000000000002e-05 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:44.952303+0000", 2024-03-26T21:33:00.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7237999999999998e-05 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:00.673 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:33:00.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:00 smithi067 ceph-mon[33346]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 547 op/s; 27437/263397 objects degraded (10.417%) 2024-03-26T21:33:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:33:00.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:00.802 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:00.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:00 smithi055 ceph-mon[40587]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 547 op/s; 27437/263397 objects degraded (10.417%) 2024-03-26T21:33:00.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:33:00.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:00 smithi055 ceph-mon[29406]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 547 op/s; 27437/263397 objects degraded (10.417%) 2024-03-26T21:33:00.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:33:00.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:00.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:00.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:00.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:01.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:01.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:01.161 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:01.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T21:33:01.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:01.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:01.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:01.448 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:01.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[29406]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[40587]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:01.653 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:01.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:33:01.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:01.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:01.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:01.907 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:01.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3865130814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:33:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:01 smithi067 ceph-mon[33346]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T21:33:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:01.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:02.006 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:02.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:02.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:02.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:02.182 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:02.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:02.325 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:02.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:02.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:02.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:02.610 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:02.680 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:02 smithi067 ceph-mon[33346]: pgmap v170: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 779 op/s; 27437/259101 objects degraded (10.589%) 2024-03-26T21:33:02.680 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 27437/263397 objects degraded (10.417%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:02.680 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:02 smithi067 ceph-mon[33346]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:33:02.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[40587]: pgmap v170: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 779 op/s; 27437/259101 objects degraded (10.589%) 2024-03-26T21:33:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 27437/263397 objects degraded (10.417%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[40587]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:33:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[29406]: pgmap v170: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 779 op/s; 27437/259101 objects degraded (10.589%) 2024-03-26T21:33:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 27437/263397 objects degraded (10.417%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:02.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:02 smithi055 ceph-mon[29406]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T21:33:02.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:02.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:02.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:02.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:03.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:03.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:03.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:03.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:03.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:03.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:03.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:03.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:03.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:03.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.685 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:03.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:03 smithi067 ceph-mon[33346]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:33:03.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:03.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:03 smithi055 ceph-mon[40587]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:33:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:03 smithi055 ceph-mon[29406]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:33:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.830 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:03.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:03.925 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:03.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:03.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:03.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:04.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:04.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:04.166 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:33:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:33:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-26T21:33:04.166 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:33:04.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:04.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:04.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:04.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:04.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:04.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:04.588 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:33:04.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:04 smithi067 ceph-mon[33346]: pgmap v172: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 962 op/s; 24039/249354 objects degraded (9.641%); 17 MiB/s, 278 objects/s recovering 2024-03-26T21:33:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3764622595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:04 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[40587]: pgmap v172: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 962 op/s; 24039/249354 objects degraded (9.641%); 17 MiB/s, 278 objects/s recovering 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3764622595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[29406]: pgmap v172: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 962 op/s; 24039/249354 objects degraded (9.641%); 17 MiB/s, 278 objects/s recovering 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3764622595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.835 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:04 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:33:04.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:33:04.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:04.912 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:04.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:05.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:05.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:05.176 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:05.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:05.308 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:05.410 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.454 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:05.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:05.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.598 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:05.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:05.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:05.737 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:05.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:05.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:05.951 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:05 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:05.951 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:05 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:33:05.951 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:05 smithi067 ceph-mon[33346]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:33:05.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[40587]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:33:06.034 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:05 smithi055 ceph-mon[29406]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T21:33:06.105 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:06.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.302 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:06.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:06.462 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:06.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:06.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:06.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[40587]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 24039/249354 objects degraded (9.641%); 20 MiB/s, 320 objects/s recovering 2024-03-26T21:33:06.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:33:06.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:33:06.758 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 24039/249354 objects degraded (9.641%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[40587]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[29406]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 24039/249354 objects degraded (9.641%); 20 MiB/s, 320 objects/s recovering 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 24039/249354 objects degraded (9.641%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:06.759 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:06 smithi055 ceph-mon[29406]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:33:06.773 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:06.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:06.898 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:33:06.923 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:06.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:06.941 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:33:06.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:06.974 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-mon[33346]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 24039/249354 objects degraded (9.641%); 20 MiB/s, 320 objects/s recovering 2024-03-26T21:33:06.974 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:33:06.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:33:06.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 24039/249354 objects degraded (9.641%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:06.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:06 smithi067 ceph-mon[33346]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T21:33:06.983 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:33:07.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:07.033 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:33:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.069 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:07.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:07.077 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:33:07.119 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:33:07.163 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:33:07.170 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:07.209 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:33:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.211 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:07.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:33:07.212 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:33:07.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:07.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:07.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:07.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:07.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.638 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:07.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:07.688 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:33:07.688 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:33:07.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:07.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:07.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3726978013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:33:07.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3726978013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:07.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:07.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:07.927 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:07.927 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:07 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:33:07.928 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:07 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:33:07.928 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3726978013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:08.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:08.054 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:08.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:33:08.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:08.208 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:08.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:08.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:08.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:08.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:08.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:33:08.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:08.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:08.790 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:08.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:08 smithi055 ceph-mon[29406]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s; 20810/235344 objects degraded (8.842%); 40 MiB/s, 646 objects/s recovering 2024-03-26T21:33:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:08 smithi055 ceph-mon[40587]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s; 20810/235344 objects degraded (8.842%); 40 MiB/s, 646 objects/s recovering 2024-03-26T21:33:08.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:08.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:08 smithi067 ceph-mon[33346]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.32k op/s; 20810/235344 objects degraded (8.842%); 40 MiB/s, 646 objects/s recovering 2024-03-26T21:33:09.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:09.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:09.163 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:09.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:09.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:09.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.337 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:09.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:09.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:09.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:09.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:09.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:09.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.744 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:09.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:09.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:09.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:10.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:10.052 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:10.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.182 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:10.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:10.250 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:10.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:33:10.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.395 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:10.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:10.496 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.542 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:10.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:10.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:10.674 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:10.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:10.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:10.947 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:10.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:10 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:33:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:10 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:33:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:10 smithi067 ceph-mon[33346]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 905 op/s; 20810/235344 objects degraded (8.842%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:33:11.049 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:33:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:33:11.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[29406]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 905 op/s; 20810/235344 objects degraded (8.842%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:33:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:33:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:33:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:10 smithi055 ceph-mon[40587]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 905 op/s; 20810/235344 objects degraded (8.842%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:33:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:11.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:11.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:11.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.367 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:11.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:11.511 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:11.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:11.648 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T21:33:11.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:11.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:11.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:11.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:11.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:11 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/756126933' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 20810/235344 objects degraded (8.842%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:12.009 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/756126933' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:12.009 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:12.009 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 20810/235344 objects degraded (8.842%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:12.010 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/756126933' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:12.010 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:33:12.010 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 20810/235344 objects degraded (8.842%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:12.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:12.075 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:12.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:12.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:12.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:12.328 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:12.338 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T21:33:12.429 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:12.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:12.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:12.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.796 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:12.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:12.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:12.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:12.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:12.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:12 smithi067 ceph-mon[33346]: pgmap v178: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 756 op/s; 20810/231216 objects degraded (9.000%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:33:12.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:12 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:33:12.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:12 smithi067 ceph-mon[33346]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:33:13.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[40587]: pgmap v178: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 756 op/s; 20810/231216 objects degraded (9.000%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[40587]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[29406]: pgmap v178: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 756 op/s; 20810/231216 objects degraded (9.000%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:33:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:12 smithi055 ceph-mon[29406]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T21:33:13.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:13.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.197 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:13.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:13.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:13.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:13.392 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:33:13.392 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:33:13.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:13.481 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:13.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:13.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:13.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:13.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:13.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:13.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:13.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:13 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:33:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:13 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:33:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:13 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:33:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:13 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:33:14.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:33:14.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:33:14.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:33:14.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:13 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:33:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:14.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:14.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:14.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:14.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:33:14.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:14.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:14.483 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:14.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:14.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:14.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:14 smithi067 ceph-mon[33346]: pgmap v180: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 17746/222246 objects degraded (7.985%); 31 MiB/s, 494 objects/s recovering 2024-03-26T21:33:14.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:14 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:33:14.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:14 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:33:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:14.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:14.760 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:14.817 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:14.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:14.940 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[40587]: pgmap v180: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 17746/222246 objects degraded (7.985%); 31 MiB/s, 494 objects/s recovering 2024-03-26T21:33:14.940 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:33:14.940 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:33:14.942 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[29406]: pgmap v180: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 17746/222246 objects degraded (7.985%); 31 MiB/s, 494 objects/s recovering 2024-03-26T21:33:14.942 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:33:14.942 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:14 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:33:14.956 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:14.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:15.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.095 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:15.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:15.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:15.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:15.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:15.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:15.501 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:15.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:15.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:15.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:15.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:15.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:15.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:15.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:15 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:33:15.973 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:15 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:33:15.974 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:15 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:33:15.974 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:15 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:33:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:15 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:33:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:15 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:33:16.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:16.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:16.170 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:16.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:16.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:16.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:16.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:16.559 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:33:16.559 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:33:16.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:16.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:16.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:16.807 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:16.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:16.891 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: pgmap v181: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 760 op/s; 17746/222246 objects degraded (7.985%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 17746/222246 objects degraded (7.985%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:16.892 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:16 smithi067 ceph-mon[33346]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:33:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:16.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:16.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:17.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: pgmap v181: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 760 op/s; 17746/222246 objects degraded (7.985%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:33:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:33:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:33:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 17746/222246 objects degraded (7.985%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[40587]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:33:17.070 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: pgmap v181: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 760 op/s; 17746/222246 objects degraded (7.985%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:33:17.070 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:33:17.070 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:33:17.070 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:17.070 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:17.071 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 17746/222246 objects degraded (7.985%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:17.071 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:16 smithi055 ceph-mon[29406]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T21:33:17.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:17.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:17.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:17.314 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:17.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:33:17.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.456 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:17.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:17.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:17.601 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:17.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:17.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:17.791 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:33:17.792 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:17 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:33:17.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:17.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:17.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:17.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.024 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:18.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:18.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:17 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:33:18.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:17 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:33:18.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:18.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:17 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:33:18.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:17 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:33:18.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:18.201 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:18.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:18.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:18.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:18.499 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:33:18.546 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:33:18.567 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:18.588 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:33:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:18.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:18.640 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:33:18.681 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:33:18.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:18.735 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: pgmap v183: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 14674/208194 objects degraded (7.048%); 31 MiB/s, 503 objects/s recovering 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:33:18.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:33:18.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:33:18.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:33:18.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2341785485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:18.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:18.755 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4230508126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:18.768 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:18.791 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:33:18.797 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:33:18.848 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:33:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: pgmap v183: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 14674/208194 objects degraded (7.048%); 31 MiB/s, 503 objects/s recovering 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2341785485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:18.862 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4230508126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:18.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:18.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:18.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:19.016 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: pgmap v183: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 14674/208194 objects degraded (7.048%); 31 MiB/s, 503 objects/s recovering 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2341785485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:19.028 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4230508126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:19.056 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:19.113 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:19.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:19.199 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:19.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:19.332 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:19.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:19.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:19.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:19.607 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:19.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:19.717 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:33:19.717 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:33:19.717 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:33:19.718 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:33:19.718 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:33:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:19.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:19.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:19.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:19.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:19.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:33:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:19 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:33:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:20.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:33:20.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T21:33:20.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:19 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:33:20.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:20.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:20.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:20.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:20.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:20.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:33:20.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:20.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:20.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:20.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:33:20.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:20.866 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:20.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:20.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:21.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[40587]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 781 op/s; 14674/208194 objects degraded (7.048%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[29406]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 781 op/s; 14674/208194 objects degraded (7.048%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:33:21.055 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:20 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:33:21.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:21.123 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:20 smithi067 ceph-mon[33346]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 781 op/s; 14674/208194 objects degraded (7.048%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:21.123 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:20 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:33:21.124 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:20 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:33:21.160 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:21.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:21.300 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:21.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:21.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:21.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:21.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:21.706 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:21.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:33:21.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:21.839 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:21.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:21.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:21.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:22.049 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:21 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:33:22.049 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:21 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:33:22.050 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:21 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 14674/208194 objects degraded (7.048%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:22.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:33:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:33:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 14674/208194 objects degraded (7.048%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:22.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:33:22.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:33:22.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:21 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 14674/208194 objects degraded (7.048%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:22.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:22.122 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:22.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:22.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:22.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:22.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:22.574 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:22.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:22.760 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.789 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151347 7.1 7.a9a74ce1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.309049367, 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672471500000001, 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151347 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.790 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484851+0000", 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6699999999999998e-06 2024-03-26T21:33:22.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484856+0000", 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7890000000000002e-06 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484864+0000", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9589999999999995e-06 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484873+0000", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1759999999999999e-06 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506221+0000", 2024-03-26T21:33:22.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021347940999999999 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506230+0000", 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0170000000000002e-06 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451562+0000", 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.945331378 2024-03-26T21:33:22.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451574+0000", 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1785e-05 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151351 7.1 7.ac3a4ac1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.309013496999999, 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672367400000003, 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151351 2024-03-26T21:33:22.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484886+0000", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5069999999999999e-06 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484889+0000", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2720000000000002e-06 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484895+0000", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8239999999999996e-06 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484901+0000", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7030000000000003e-06 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506254+0000", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021352684 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506260+0000", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8440000000000003e-06 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451599+0000", 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94533952600000004 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451608+0000", 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3139999999999993e-06 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151354 7.1 7.e5ae2af1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.308989373999999, 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672899499999998, 2024-03-26T21:33:22.800 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151354 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484911+0000", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.17e-06 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484916+0000", 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6129999999999997e-06 2024-03-26T21:33:22.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484920+0000", 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9639999999999997e-06 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484926+0000", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000002e-06 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506282+0000", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021356001999999999 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506287+0000", 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6210000000000001e-06 2024-03-26T21:33:22.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451630+0000", 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94534287400000006 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451638+0000", 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6709999999999999e-06 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151355 7.1 7.e69c8861 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.308965626999999, 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96673204199999996, 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151355 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484935+0000", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0640000000000001e-06 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484938+0000", 2024-03-26T21:33:22.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4010000000000001e-06 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484942+0000", 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.459e-06 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484947+0000", 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9609999999999996e-06 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506313+0000", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021365129 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506319+0000", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.409e-06 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451657+0000", 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94533827999999998 2024-03-26T21:33:22.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451665+0000", 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151366 7.1 7.72230521 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.499642+0000", 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.294256664999999, 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95205036499999995, 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151366 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499642+0000", 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499642+0000", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499643+0000", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.502e-06 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499648+0000", 2024-03-26T21:33:22.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7419999999999997e-06 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499658+0000", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0115000000000001e-05 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.499674+0000", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6069000000000001e-05 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506339+0000", 2024-03-26T21:33:22.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066654490000000004 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506343+0000", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4359999999999998e-06 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451684+0000", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.945341298 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451692+0000", 2024-03-26T21:33:22.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7540000000000006e-06 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151376 7.1 7.134db381 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.519217+0000", 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.274680702000001, 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93250306199999999, 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:33:22.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151376 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519217+0000", 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519217+0000", 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519219+0000", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6619999999999999e-06 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519227+0000", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8939999999999993e-06 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519243+0000", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5608999999999999e-05 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519251+0000", 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1939999999999998e-06 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519303+0000", 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1709000000000002e-05 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.519309+0000", 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7490000000000001e-06 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451713+0000", 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93240365999999997 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451721+0000", 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5850000000000002e-06 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.822 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1422 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94208173344896 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.934703+0000", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.859195280000002, 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.54863800799999995, 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1422 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934703+0000", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934703+0000", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934704+0000", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1000000000000001e-06 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934708+0000", 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2729999999999996e-06 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934717+0000", 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6589999999999996e-06 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934726+0000", 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5890000000000003e-06 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934778+0000", 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2246000000000001e-05 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.934786+0000", 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.480823+0000", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.54603630599999997 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.480957+0000", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013456899999999999 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.483341+0000", 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002383834 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151347 7.1 e135/134 7:8732e595:::benchmark_data_smithi055_81491_object31860:head v 135'9335, mlcod=0'0)", 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.656876+0000", 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.137022559999998, 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017907217, 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656876+0000", 2024-03-26T21:33:22.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656876+0000", 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656876+0000", 2024-03-26T21:33:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656882+0000", 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7830000000000004e-06 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656893+0000", 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1072e-05 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656902+0000", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9099999999999994e-06 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656961+0000", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8816000000000001e-05 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.656985+0000", 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.374e-05 2024-03-26T21:33:22.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.658754+0000", 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001769453 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674783+0000", 2024-03-26T21:33:22.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016028730000000001 2024-03-26T21:33:22.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151351 7.1 e135/134 7:83525c35:::benchmark_data_smithi055_81491_object31864:head v 135'9336, mlcod=0'0)", 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.660142+0000", 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.133756128000002, 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019182187, 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660142+0000", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660142+0000", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660143+0000", 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.06e-07 2024-03-26T21:33:22.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660148+0000", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5149999999999997e-06 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660157+0000", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8780000000000004e-06 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660175+0000", 2024-03-26T21:33:22.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7397e-05 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674964+0000", 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014789490000000001 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674984+0000", 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0228999999999999e-05 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.679257+0000", 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042728269999999999 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.679324+0000", 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7144999999999995e-05 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(7.3 135/134 [PushOp(7:c001b81c:::benchmark_data_smithi055_81491_object82524:head, version: 115'5151, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:c001b81c:::benchmark_data_smithi055_81491_object82524:head@115'5151, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.660218+0000", 2024-03-26T21:33:22.838 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.133679730000001, 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014918984999999999, 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660218+0000", 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660218+0000", 2024-03-26T21:33:22.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.660235+0000", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6991e-05 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674655+0000", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014419342999999999 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674676+0000", 2024-03-26T21:33:22.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1240999999999999e-05 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674692+0000", 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5897999999999999e-05 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674849+0000", 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015680199999999999 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674862+0000", 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2921999999999999e-05 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.675137+0000", 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027578800000000001 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151359 7.9 e135/134 7:9c99c2eb:::benchmark_data_smithi055_81491_object31872:head v 135'9698, mlcod=0'0)", 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.665229+0000", 2024-03-26T21:33:22.842 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.128669331000001, 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015572753999999999, 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.665229+0000", 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.665229+0000", 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.665237+0000", 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3049999999999993e-06 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.665245+0000", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3380000000000005e-06 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.665266+0000", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0579000000000001e-05 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674587+0000", 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093204429999999994 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674902+0000", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031552099999999998 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674921+0000", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8593999999999998e-05 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.680762+0000", 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058415660000000003 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.680802+0000", 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9407999999999999e-05 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151354 7.1 e135/134 7:8f5475a7:::benchmark_data_smithi055_81491_object31867:head v 135'9337, mlcod=0'0)", 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.674766+0000", 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.119132573000002, 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010603458, 2024-03-26T21:33:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674766+0000", 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674766+0000", 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674766+0000", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7899999999999999e-07 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674771+0000", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2569999999999998e-06 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674779+0000", 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3250000000000002e-06 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.674785+0000", 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.443e-06 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.679358+0000", 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045731510000000001 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.679379+0000", 2024-03-26T21:33:22.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-03-26T21:33:22.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.685329+0000", 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059490639999999996 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.685369+0000", 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0553000000000003e-05 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151458 7.8 e135/134 7:1322f680:::benchmark_data_smithi055_81491_object31971:head v 135'9618, mlcod=135'9616)", 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.741955+0000", 2024-03-26T21:33:22.852 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.051942952000001, 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028315185, 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741955+0000", 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741955+0000", 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741956+0000", 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.77e-07 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741967+0000", 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0976e-05 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741975+0000", 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8570000000000001e-06 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741981+0000", 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2050000000000001e-06 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.742065+0000", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4400000000000005e-05 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.742084+0000", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9417000000000001e-05 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770226+0000", 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028142019000000001 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770270+0000", 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3934000000000003e-05 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(7.8 135/134 [PushOp(7:1003669a:::benchmark_data_smithi055_81491_object116819:head, version: 121'7386, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1003669a:::benchmark_data_smithi055_81491_object116819:head@121'7386, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.741991+0000", 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.051907548999999, 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028774826, 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.857 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741991+0000", 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741991+0000", 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.741991+0000", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4200000000000002e-07 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.742062+0000", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0823999999999998e-05 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.742084+0000", 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1831000000000002e-05 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.742098+0000", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4459e-05 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770520+0000", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028422095000000001 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770535+0000", 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4440000000000001e-05 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770765+0000", 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023093500000000001 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_backfill(progress 7.4 e 135/135 lb 7:200b10a8:::benchmark_data_smithi055_81491_object88398:head)", 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.743608+0000", 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.050290660999998, 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025718474000000002, 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743608+0000", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743608+0000", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743608+0000", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5700000000000002e-07 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743615+0000", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.691e-06 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743623+0000", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0059999999999992e-06 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743628+0000", 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3900000000000001e-06 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743684+0000", 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6020000000000002e-05 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769262+0000", 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025577872000000001 2024-03-26T21:33:22.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769326+0000", 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3837999999999994e-05 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.865 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151457 7.4 e135/134 7:2766624e:::benchmark_data_smithi055_81491_object31970:head v 135'9456, mlcod=135'9454)", 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.743642+0000", 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.050255863, 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028945285000000001, 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743642+0000", 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743642+0000", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743643+0000", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.22e-07 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743653+0000", 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0567000000000001e-05 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743658+0000", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7219999999999999e-06 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743663+0000", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6779999999999999e-06 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769373+0000", 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025710670000000001 2024-03-26T21:33:22.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769396+0000", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2276000000000002e-05 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.772551+0000", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031558659999999998 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.772588+0000", 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6183999999999998e-05 2024-03-26T21:33:22.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(7.4 135/134 [PushOp(7:200bf9a9:::benchmark_data_smithi055_81491_object115441:head, version: 119'7217, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:200bf9a9:::benchmark_data_smithi055_81491_object115441:head@119'7217, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.743671+0000", 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.050226748, 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027870492, 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743671+0000", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743671+0000", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743672+0000", 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2999999999999999e-07 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.743725+0000", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3341000000000003e-05 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769195+0000", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025469518 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.769229+0000", 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4221000000000003e-05 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.771319+0000", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020898259999999999 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.771331+0000", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2262000000000001e-05 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.771542+0000", 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211094 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:151467 7.3 e135/134 7:cd2a1622:::benchmark_data_smithi055_81491_object31980:head v 135'9499, mlcod=135'9497)", 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:56.745679+0000", 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.048219535000001, 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029387670000000001, 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.745679+0000", 2024-03-26T21:33:22.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.745679+0000", 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.745679+0000", 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000005e-07 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.745684+0000", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4610000000000001e-06 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770104+0000", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024420049999999999 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770114+0000", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0294e-05 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770166+0000", 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805000000000003e-05 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.770186+0000", 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0078e-05 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.774995+0000", 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048087629999999998 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:56.775066+0000", 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1594000000000006e-05 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:155405 7.8 e136/134 7:12148e6a:::benchmark_data_smithi055_81491_object35918:head v 136'9867, mlcod=136'9864)", 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:00.851948+0000", 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.941949726000001, 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010347688000000001, 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851948+0000", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851948+0000", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851949+0000", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8999999999999995e-07 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851955+0000", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8590000000000001e-06 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851966+0000", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1182999999999999e-05 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.851973+0000", 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8770000000000004e-06 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.852070+0000", 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6910999999999994e-05 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.852109+0000", 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8454000000000001e-05 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.862228+0000", 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010119415999999999 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:00.862296+0000", 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8198e-05 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.24704.0:172644 7.4 e143/134 7:263d8354:::benchmark_data_smithi055_81491_object53157:head v 143'10758, mlcod=143'10756)", 2024-03-26T21:33:22.882 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:19.600181+0000", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.1937169760000002, 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01066782, 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600181+0000", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600181+0000", 2024-03-26T21:33:22.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600182+0000", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5000000000000002e-07 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600187+0000", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4040000000000003e-06 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600195+0000", 2024-03-26T21:33:22.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1240000000000005e-06 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600202+0000", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4790000000000001e-06 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600455+0000", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271700000000001 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.600485+0000", 2024-03-26T21:33:22.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0593999999999999e-05 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.610792+0000", 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010307217 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:19.610849+0000", 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6635000000000002e-05 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:33:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:33:22.887 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:33:22.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:22.902 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:22 smithi055 ceph-mon[40587]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 14674/203721 objects degraded (7.203%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:22.902 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:22 smithi055 ceph-mon[40587]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:33:22.903 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:22 smithi055 ceph-mon[29406]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 14674/203721 objects degraded (7.203%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:22.903 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:22 smithi055 ceph-mon[29406]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:33:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:22.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:22.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:22 smithi067 ceph-mon[33346]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 14674/203721 objects degraded (7.203%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:33:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:22 smithi067 ceph-mon[33346]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T21:33:23.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:23.077 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:23.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:23.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:23.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:23.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:23.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:23.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:23.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:23.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:23.769 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:23.817 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T21:33:23.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:23.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:23.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:23.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:33:24.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:24.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2366268849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:33:24.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:33:24.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:33:24.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:33:24.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:33:24.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2366268849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:23 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:24.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:33:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:33:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:33:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:33:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:33:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:33:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2366268849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:23 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:33:24.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:24.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:33:24.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:24.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:24.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:24.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T21:33:24.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:24.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:24.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:24.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:24.988 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:24.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 11653/193884 objects degraded (6.010%); 22 MiB/s, 350 objects/s recovering 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: 5.2 deep-scrub starts 2024-03-26T21:33:25.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[40587]: 5.2 deep-scrub ok 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 11653/193884 objects degraded (6.010%); 22 MiB/s, 350 objects/s recovering 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: 5.2 deep-scrub starts 2024-03-26T21:33:25.063 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:24 smithi055 ceph-mon[29406]: 5.2 deep-scrub ok 2024-03-26T21:33:25.090 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:25.148 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:25.150 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 11653/193884 objects degraded (6.010%); 22 MiB/s, 350 objects/s recovering 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: 5.2 deep-scrub starts 2024-03-26T21:33:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:24 smithi067 ceph-mon[33346]: 5.2 deep-scrub ok 2024-03-26T21:33:25.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:25.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.413 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:25.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:25.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:25.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:25.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:25.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:25.936 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:25.936 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:25.948 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:25.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:26.048 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:33:26.048 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:33:26.048 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[29406]: 4.a deep-scrub starts 2024-03-26T21:33:26.048 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[29406]: 4.a deep-scrub ok 2024-03-26T21:33:26.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:33:26.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:33:26.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[40587]: 4.a deep-scrub starts 2024-03-26T21:33:26.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:25 smithi055 ceph-mon[40587]: 4.a deep-scrub ok 2024-03-26T21:33:26.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:26.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:26.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:25 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:33:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:25 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:33:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:25 smithi067 ceph-mon[33346]: 4.a deep-scrub starts 2024-03-26T21:33:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:25 smithi067 ceph-mon[33346]: 4.a deep-scrub ok 2024-03-26T21:33:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.257 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:26.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:26.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:26.388 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:26.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:26.528 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:26.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:26.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:26.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:26.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:26.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:26.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:26.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:26.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:26 smithi067 ceph-mon[33346]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 777 B/s rd, 0 B/s wr, 724 op/s; 11653/193884 objects degraded (6.010%); 20 MiB/s, 319 objects/s recovering 2024-03-26T21:33:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:26 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:33:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:26 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:33:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:26 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 11653/193884 objects degraded (6.010%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:26.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:27.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:27.061 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[29406]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 777 B/s rd, 0 B/s wr, 724 op/s; 11653/193884 objects degraded (6.010%); 20 MiB/s, 319 objects/s recovering 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 11653/193884 objects degraded (6.010%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[40587]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 777 B/s rd, 0 B/s wr, 724 op/s; 11653/193884 objects degraded (6.010%); 20 MiB/s, 319 objects/s recovering 2024-03-26T21:33:27.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:33:27.063 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:33:27.063 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:26 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 11653/193884 objects degraded (6.010%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:27.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:27.106 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:33:27.151 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:33:27.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:27.207 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:33:27.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:27.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:33:27.310 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:33:27.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:27.364 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:33:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:27.415 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:33:27.420 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:33:27.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:27.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:27.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.773 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:27.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:27.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:27.923 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:27.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:27.943 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1099831578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:33:27.944 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1099831578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:28.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:28.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:28.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:27 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:33:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:27 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:33:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1099831578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:28.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:28.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:28.445 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:28.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:28.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:28.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:28.728 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:28.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:28.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:28.970 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:29.005 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:29.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.140 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:29.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:28 smithi067 ceph-mon[33346]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8859/180024 objects degraded (4.921%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:33:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:28 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:33:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:28 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:33:29.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:29.268 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:29.297 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[29406]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8859/180024 objects degraded (4.921%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[40587]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8859/180024 objects degraded (4.921%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:33:29.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:28 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:33:29.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:29.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:29.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:29.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:29.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:29.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:29.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:29.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:29.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:29.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:29.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:30.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.041 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:30.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[40587]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:33:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:29 smithi055 ceph-mon[29406]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:33:30.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:29 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:33:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:29 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:33:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:29 smithi067 ceph-mon[33346]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T21:33:30.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:30.297 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:30.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.423 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:30.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:30.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:30.551 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:30.653 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:30.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:30.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:30.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:30.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:30.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:30.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:33:31.035 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:31.152 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 610 op/s; 8859/180024 objects degraded (4.921%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 4.15 deep-scrub starts 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: 4.15 deep-scrub ok 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:31.153 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 610 op/s; 8859/180024 objects degraded (4.921%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 4.15 deep-scrub starts 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: 4.15 deep-scrub ok 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:31.154 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:31.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 610 op/s; 8859/180024 objects degraded (4.921%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 4.15 deep-scrub starts 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: 4.15 deep-scrub ok 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:31.246 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:31.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:31.507 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.559 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:31.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:31.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:31.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:31.777 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:31.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.816 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:31.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:31.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:31.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:32.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:32.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:33:32.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 8859/180024 objects degraded (4.921%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2458799618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:33:32.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 8859/180024 objects degraded (4.921%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2458799618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:31 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:32.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:33:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:33:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:33:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:33:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 8859/180024 objects degraded (4.921%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2458799618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:31 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:33:32.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:32.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.478 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:32.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:32.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:32.603 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T21:33:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:32.622 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:33:32.700 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:33:32.701 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:33:32.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:32.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:32.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:32.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:32.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.016 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:33.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:33.117 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:33.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: pgmap v197: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 934 op/s; 8859/177096 objects degraded (5.002%); 21 MiB/s, 340 objects/s recovering 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2778897895' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:33:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:33:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:32 smithi067 ceph-mon[33346]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:33:33.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:33.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: pgmap v197: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 934 op/s; 8859/177096 objects degraded (5.002%); 21 MiB/s, 340 objects/s recovering 2024-03-26T21:33:33.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:33:33.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:33:33.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2778897895' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[40587]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: pgmap v197: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 934 op/s; 8859/177096 objects degraded (5.002%); 21 MiB/s, 340 objects/s recovering 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2778897895' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:33:33.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:33:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:32 smithi055 ceph-mon[29406]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T21:33:33.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:33.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:33.521 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:33.566 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:33.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:33.798 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:33.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:33.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:33.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:33.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:34.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:34.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:34.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:33 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:33:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:33 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:33:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:33 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:33:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:33 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:33:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:33 smithi067 ceph-mon[33346]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:33:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:34.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:34.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[40587]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:33:34.296 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:33 smithi055 ceph-mon[29406]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T21:33:34.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:33:34.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:34.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:34.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:34.617 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:34.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:34.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:34.829 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:34.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:34.971 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:34.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:35.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T21:33:35.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:35.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:35.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:35.230 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:35.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:34 smithi055 ceph-mon[29406]: pgmap v200: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 5153/170784 objects degraded (3.017%); 52 MiB/s, 831 objects/s recovering 2024-03-26T21:33:35.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:34 smithi055 ceph-mon[29406]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:33:35.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:34 smithi055 ceph-mon[40587]: pgmap v200: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 5153/170784 objects degraded (3.017%); 52 MiB/s, 831 objects/s recovering 2024-03-26T21:33:35.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:34 smithi055 ceph-mon[40587]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:33:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:34 smithi067 ceph-mon[33346]: pgmap v200: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 771 op/s; 5153/170784 objects degraded (3.017%); 52 MiB/s, 831 objects/s recovering 2024-03-26T21:33:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:34 smithi067 ceph-mon[33346]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:33:35.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:35.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:35.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:35.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.658 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:35.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:35.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:35.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:35.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:35.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:33:35.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:35.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:36.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.061 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:36.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:36.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:36.208 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:36.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:36.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:36.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:36.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:36.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:36.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:36.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:36.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:36.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:36.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:36.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.009 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:37.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:37.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:37.133 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:37.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:37.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:36 smithi067 ceph-mon[33346]: pgmap v202: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 691 op/s; 5153/170784 objects degraded (3.017%); 47 MiB/s, 745 objects/s recovering 2024-03-26T21:33:37.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:36 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5153/170784 objects degraded (3.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:37.275 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:37.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:36 smithi055 ceph-mon[29406]: pgmap v202: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 691 op/s; 5153/170784 objects degraded (3.017%); 47 MiB/s, 745 objects/s recovering 2024-03-26T21:33:37.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:36 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5153/170784 objects degraded (3.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:36 smithi055 ceph-mon[40587]: pgmap v202: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 691 op/s; 5153/170784 objects degraded (3.017%); 47 MiB/s, 745 objects/s recovering 2024-03-26T21:33:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:36 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5153/170784 objects degraded (3.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:37.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:37.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:37.551 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:37.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:37.683 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:37.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:37.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:37.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:37.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:37.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:38.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:38.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:38.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:38.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:37 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:38.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:38.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:38.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:37 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:38.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:38.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:37 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:33:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.333 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:38.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:38.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:38.473 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:38.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:38.598 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:38.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.764 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:38.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:38.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:38.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:38.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:39.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:39.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:39.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:39.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:38 smithi067 ceph-mon[33346]: pgmap v203: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 3455/156570 objects degraded (2.207%); 48 MiB/s, 773 objects/s recovering 2024-03-26T21:33:39.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:39.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:38 smithi055 ceph-mon[29406]: pgmap v203: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 3455/156570 objects degraded (2.207%); 48 MiB/s, 773 objects/s recovering 2024-03-26T21:33:39.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:38 smithi055 ceph-mon[40587]: pgmap v203: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 3455/156570 objects degraded (2.207%); 48 MiB/s, 773 objects/s recovering 2024-03-26T21:33:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:39.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:39.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:39.463 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:39.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:39.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.737 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:39.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:39.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:39.864 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:39.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:39.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:40.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:40.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:40.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:40.260 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:40.362 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:40.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:40.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:40.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:40.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:40.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:40.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:40.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:40.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:40.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:41.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:41.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:41.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:41.258 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:41.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[40587]: pgmap v204: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 658 B/s rd, 0 B/s wr, 805 op/s; 3455/156570 objects degraded (2.207%); 34 MiB/s, 547 objects/s recovering 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[29406]: pgmap v204: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 658 B/s rd, 0 B/s wr, 805 op/s; 3455/156570 objects degraded (2.207%); 34 MiB/s, 547 objects/s recovering 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:41 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:41.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:41.389 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:41 smithi067 ceph-mon[33346]: pgmap v204: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 658 B/s rd, 0 B/s wr, 805 op/s; 3455/156570 objects degraded (2.207%); 34 MiB/s, 547 objects/s recovering 2024-03-26T21:33:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:41 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:33:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:41 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:33:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:41 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:41.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:41.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.665 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:41.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:41.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.794 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:41.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:41.807 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:41.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:41.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:42.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:42.058 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:42.140 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:42.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:42.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:42.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 3455/156570 objects degraded (2.207%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:42.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 3455/156570 objects degraded (2.207%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:42.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:42.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:42.393 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:42.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 3455/156570 objects degraded (2.207%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:42.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:42.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.651 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:42.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:42.738 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:33:42.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:42.787 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:33:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:42.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:42.834 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:33:42.876 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:33:42.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:42.920 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:33:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:42.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:42.956 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:33:42.998 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:33:43.021 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:43.040 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:33:43.041 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:43 smithi055 ceph-mon[29406]: pgmap v205: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 3455/152907 objects degraded (2.260%); 9.4 MiB/s, 149 objects/s recovering 2024-03-26T21:33:43.041 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3232158907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:43.041 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:43 smithi055 ceph-mon[40587]: pgmap v205: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 3455/152907 objects degraded (2.260%); 9.4 MiB/s, 149 objects/s recovering 2024-03-26T21:33:43.041 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3232158907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:43.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:33:43.073 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:33:43.074 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:33:43.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:43.208 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:43.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:43.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:43 smithi067 ceph-mon[33346]: pgmap v205: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 3455/152907 objects degraded (2.260%); 9.4 MiB/s, 149 objects/s recovering 2024-03-26T21:33:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3232158907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:43.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:43.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:43.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:43.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:43.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:43.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:43.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:44.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:44.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:44.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:44.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:44.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:44.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:44.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:44.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:44.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:44.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:44.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:44.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:44.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:44.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.025 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:45.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:45.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:45.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:45.283 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:45 smithi055 ceph-mon[40587]: pgmap v206: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s; 2139/144534 objects degraded (1.480%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:45.283 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:45 smithi055 ceph-mon[29406]: pgmap v206: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s; 2139/144534 objects degraded (1.480%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:45.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:45.434 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:45.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:45 smithi067 ceph-mon[33346]: pgmap v206: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s; 2139/144534 objects degraded (1.480%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:45.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:45.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:45.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:45.824 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:45.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:45.958 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:45.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:46.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:46.098 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:46.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:46.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:46.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:46.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:46.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:46.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:46.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:46.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:46.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:46.902 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:46.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:46.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:46 smithi067 ceph-mon[33346]: pgmap v207: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 743 op/s; 2139/144534 objects degraded (1.480%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:33:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:46 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2139/144534 objects degraded (1.480%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:47.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:47.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[40587]: pgmap v207: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 743 op/s; 2139/144534 objects degraded (1.480%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:33:47.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:47.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:47.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2139/144534 objects degraded (1.480%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[29406]: pgmap v207: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 869 B/s rd, 0 B/s wr, 743 op/s; 2139/144534 objects degraded (1.480%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:33:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:33:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:33:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:46 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2139/144534 objects degraded (1.480%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:47.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:47.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:47.385 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:47.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:47.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:47.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:47.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:47.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:47.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:47.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:47.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:48.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:48.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:48.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:48.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:48.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.357 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:48.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:48.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:48.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:48.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:48.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:48.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:48.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:48.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:48.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:48.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:48.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:49.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:48 smithi055 ceph-mon[29406]: pgmap v208: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 664/129939 objects degraded (0.511%); 19 MiB/s, 311 objects/s recovering 2024-03-26T21:33:49.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:49.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:48 smithi055 ceph-mon[40587]: pgmap v208: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 664/129939 objects degraded (0.511%); 19 MiB/s, 311 objects/s recovering 2024-03-26T21:33:49.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:49.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:49.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:48 smithi067 ceph-mon[33346]: pgmap v208: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 664/129939 objects degraded (0.511%); 19 MiB/s, 311 objects/s recovering 2024-03-26T21:33:49.265 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:49.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.393 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:49.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:49.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:49.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:49.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:49.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:49.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:49.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:49.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:49.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:49.917 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:49 smithi055 ceph-mon[29406]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:33:49.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:50.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:50.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:50.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:50.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:49 smithi067 ceph-mon[33346]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:33:50.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:49 smithi055 ceph-mon[40587]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T21:33:50.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:50.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:50.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:50.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:50.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.551 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:50.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:50.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:50.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.802 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:50.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:50.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:50.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:50.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:51.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:51.036 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:50 smithi055 ceph-mon[40587]: pgmap v210: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s; 664/129939 objects degraded (0.511%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:51.036 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:50 smithi055 ceph-mon[40587]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:33:51.037 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:50 smithi055 ceph-mon[29406]: pgmap v210: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s; 664/129939 objects degraded (0.511%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:51.037 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:50 smithi055 ceph-mon[29406]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:33:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:51.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:51.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:50 smithi067 ceph-mon[33346]: pgmap v210: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s; 664/129939 objects degraded (0.511%); 15 MiB/s, 241 objects/s recovering 2024-03-26T21:33:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:50 smithi067 ceph-mon[33346]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T21:33:51.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.333 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:51.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:51.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:51.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:51.588 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:33:51.588 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:33:51.588 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:33:51.588 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:33:51.588 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:33:51.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:51.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:51.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:51.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:51.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:51.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:51.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:52.012 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:52.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:52.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[40587]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 664/129939 objects degraded (0.511%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[40587]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[29406]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 664/129939 objects degraded (0.511%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:52.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:51 smithi055 ceph-mon[29406]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:33:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:51 smithi067 ceph-mon[33346]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T21:33:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:51 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 664/129939 objects degraded (0.511%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:33:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:51 smithi067 ceph-mon[33346]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T21:33:52.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:52.283 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:52.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:52.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:52.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:52.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:52.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:52.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:52.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:52.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:52.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:52.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:52.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:52.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:52.978 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:33:52.984 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T21:33:53.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:53.097 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:53.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[29406]: pgmap v213: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 953 op/s; 664/127398 objects degraded (0.521%); 14 MiB/s, 219 objects/s recovering 2024-03-26T21:33:53.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3248898863' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1312991453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[40587]: pgmap v213: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 953 op/s; 664/127398 objects degraded (0.521%); 14 MiB/s, 219 objects/s recovering 2024-03-26T21:33:53.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3248898863' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1312991453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:52 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:53.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:33:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:52 smithi067 ceph-mon[33346]: pgmap v213: 148 pgs: 2 active+undersized+degraded+remapped+backfilling, 146 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 953 op/s; 664/127398 objects degraded (0.521%); 14 MiB/s, 219 objects/s recovering 2024-03-26T21:33:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3248898863' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:52 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:33:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1312991453' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:52 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:33:53.243 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:53.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.385 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:53.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:53.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:53.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:53.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:33:53.664 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:53.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:53.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:53.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:53.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:54.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:54.073 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:33:54.163 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:33:54.164 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:53 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:33:54.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:54.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:33:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:53 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:33:54.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.357 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:54.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:54.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:54.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:54.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:54.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T21:33:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.674 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:54.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:54.703 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:54.725 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:54.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:54.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:54.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:54.985 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:54.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:55.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:55.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:55.132 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:54 smithi055 ceph-mon[29406]: pgmap v216: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 617 op/s; 199/122547 objects degraded (0.162%); 7.1 MiB/s, 112 objects/s recovering 2024-03-26T21:33:55.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:54 smithi055 ceph-mon[29406]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:33:55.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:54 smithi055 ceph-mon[40587]: pgmap v216: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 617 op/s; 199/122547 objects degraded (0.162%); 7.1 MiB/s, 112 objects/s recovering 2024-03-26T21:33:55.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:54 smithi055 ceph-mon[40587]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:33:55.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:54 smithi067 ceph-mon[33346]: pgmap v216: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 617 op/s; 199/122547 objects degraded (0.162%); 7.1 MiB/s, 112 objects/s recovering 2024-03-26T21:33:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:54 smithi067 ceph-mon[33346]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T21:33:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:55.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:55.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:55.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:55.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:55.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:55.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:55.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:55.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:55.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:55.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:33:55.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:55.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:55.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:56.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:56.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:56.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:56.259 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:56 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 199/122547 objects degraded (0.162%), 1 pg degraded) 2024-03-26T21:33:56.259 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:56 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 199/122547 objects degraded (0.162%), 1 pg degraded) 2024-03-26T21:33:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.294 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:56.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:56.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:56.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:56.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:56 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 199/122547 objects degraded (0.162%), 1 pg degraded) 2024-03-26T21:33:56.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:33:56.561 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:33:56.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:56.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:56.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:56.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:56.972 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:56.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:56.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:33:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:33:57.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:57.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:57 smithi055 ceph-mon[40587]: pgmap v218: 148 pgs: 148 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 612 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-03-26T21:33:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:57 smithi055 ceph-mon[29406]: pgmap v218: 148 pgs: 148 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 612 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-03-26T21:33:57.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:57.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:57.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:57 smithi067 ceph-mon[33346]: pgmap v218: 148 pgs: 148 active+clean; 2.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 612 op/s; 9.1 MiB/s, 145 objects/s recovering 2024-03-26T21:33:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:57.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:57.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:57.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:57.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:57.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:57.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:58.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:33:58.032 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:33:58.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:58.148 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197462 7.6 7:6b39b75d:::benchmark_data_smithi055_81491_object77975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.704612+0000", 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 9.446990606, 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.18179126400000001, 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.149 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197462 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704612+0000", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704612+0000", 2024-03-26T21:33:58.150 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704613+0000", 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.026e-06 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704617+0000", 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9020000000000001e-06 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.151 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704629+0000", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2079000000000001e-05 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704637+0000", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7910000000000005e-06 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704702+0000", 2024-03-26T21:33:58.152 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5272999999999997e-05 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704817+0000", 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011424100000000001 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704920+0000", 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103439 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.153 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.711799+0000", 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0068793539999999999 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.711837+0000", 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7919000000000001e-05 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886318+0000", 2024-03-26T21:33:58.154 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.174480844 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886365+0000", 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6468000000000003e-05 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.155 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886404+0000", 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8927999999999997e-05 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197463 7.6 7:67e6e31c:::benchmark_data_smithi055_81491_object77976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.704655+0000", 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 9.44694799, 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.18186930000000001, 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.156 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197463 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704655+0000", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.157 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704655+0000", 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704655+0000", 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2100000000000002e-07 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704658+0000", 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.5780000000000001e-06 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.158 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704668+0000", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8230000000000006e-06 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.704673+0000", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.587e-06 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.708164+0000", 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034903239999999999 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.159 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.708314+0000", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000150521 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.708432+0000", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011750400000000001 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.711936+0000", 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0035043090000000002 2024-03-26T21:33:58.160 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.714644+0000", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0027076460000000002 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886442+0000", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.171798336 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886488+0000", 2024-03-26T21:33:58.161 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5482999999999998e-05 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886524+0000", 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6767999999999998e-05 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197477 7.6 7:6c8993eb:::benchmark_data_smithi055_81491_object77990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.716258+0000", 2024-03-26T21:33:58.162 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 9.4353446559999998, 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.170384902, 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197477 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.163 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716258+0000", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716258+0000", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716259+0000", 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5299999999999999e-07 2024-03-26T21:33:58.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716261+0000", 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7199999999999998e-06 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.165 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716267+0000", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6779999999999997e-06 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.716273+0000", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4659999999999998e-06 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.718214+0000", 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0019417499999999999 2024-03-26T21:33:58.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.718328+0000", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011325200000000001 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.718447+0000", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119694 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.721809+0000", 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003361967 2024-03-26T21:33:58.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.723767+0000", 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001958151 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886563+0000", 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.162795052 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.168 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886606+0000", 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3776999999999998e-05 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886643+0000", 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6942e-05 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:197497 7.d e151/95 7:ba42a995:::benchmark_data_smithi055_81491_object78010:head v 151'12209, mlcod=151'12207)", 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.737616+0000", 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 9.4139869209999993, 2024-03-26T21:33:58.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16609067399999999, 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737616+0000", 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737616+0000", 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.170 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737617+0000", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2200000000000004e-07 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737621+0000", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5410000000000002e-06 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737628+0000", 2024-03-26T21:33:58.171 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.883e-06 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737633+0000", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.434e-06 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737698+0000", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4848e-05 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.737720+0000", 2024-03-26T21:33:58.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1888000000000001e-05 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.903619+0000", 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16589852699999999 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.903707+0000", 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8030999999999998e-05 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197736 7.9 7:974c4e52:::benchmark_data_smithi055_81491_object78249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.8495393199999999, 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.65874689099999995, 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197736 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.174 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302065+0000", 2024-03-26T21:33:58.175 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3850000000000001e-06 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302072+0000", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2169999999999997e-06 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302085+0000", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2845e-05 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302095+0000", 2024-03-26T21:33:58.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0302999999999999e-05 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302199+0000", 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103849 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302211+0000", 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1514e-05 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954115+0000", 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.65190433400000003 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954255+0000", 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000139825 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954396+0000", 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000141545 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956845+0000", 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024481049999999999 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958788+0000", 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001942924 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960742+0000", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0019545600000000002 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960775+0000", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2493000000000002e-05 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960811+0000", 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5991999999999997e-05 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197747 7.9 7:9a8570b5:::benchmark_data_smithi055_81491_object78260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.8322647830000003, 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.64169457200000002, 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.182 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197747 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319340+0000", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1229999999999998e-06 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319361+0000", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0188000000000001e-05 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319432+0000", 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1435999999999998e-05 2024-03-26T21:33:58.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319444+0000", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1871999999999999e-05 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319586+0000", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000142278 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319597+0000", 2024-03-26T21:33:58.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1341e-05 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956876+0000", 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63727853700000003 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956953+0000", 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7040999999999996e-05 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.957055+0000", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010241700000000001 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960899+0000", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038431479999999998 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960953+0000", 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4721e-05 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960962+0000", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7830000000000004e-06 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960990+0000", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7970000000000002e-05 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961033+0000", 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2716999999999998e-05 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.188 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197755 7.9 7:9cc27710:::benchmark_data_smithi055_81491_object78268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.8219899509999999, 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63164425599999996, 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197755 2024-03-26T21:33:58.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329614+0000", 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6299999999999993e-07 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329620+0000", 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7230000000000001e-06 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329630+0000", 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0569e-05 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329638+0000", 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8010000000000004e-06 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329725+0000", 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6638999999999995e-05 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329734+0000", 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2809999999999997e-06 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958820+0000", 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62908593499999998 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958885+0000", 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5277000000000005e-05 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958951+0000", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5949000000000002e-05 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961041+0000", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020901750000000001 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961088+0000", 2024-03-26T21:33:58.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6925999999999998e-05 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961168+0000", 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9276000000000005e-05 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961219+0000", 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1544e-05 2024-03-26T21:33:58.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961257+0000", 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8198000000000003e-05 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197762 7.9 7:9b3826cf:::benchmark_data_smithi055_81491_object78275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.8161049729999998, 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63713577799999999, 2024-03-26T21:33:58.196 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197762 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335499+0000", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4839999999999999e-06 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335505+0000", 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3909999999999997e-06 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335515+0000", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0103e-05 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335523+0000", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1610000000000003e-06 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335590+0000", 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6483999999999996e-05 2024-03-26T21:33:58.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335598+0000", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6160000000000002e-06 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961364+0000", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62576607200000001 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961461+0000", 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6230000000000005e-05 2024-03-26T21:33:58.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961528+0000", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7299999999999996e-05 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972488+0000", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.01096025 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972501+0000", 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.307e-05 2024-03-26T21:33:58.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972538+0000", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6834e-05 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972589+0000", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1484999999999999e-05 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972634+0000", 2024-03-26T21:33:58.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4298000000000002e-05 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197765 7.9 7:9de1617a:::benchmark_data_smithi055_81491_object78278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.816016726, 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63724420299999995, 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197765 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335588+0000", 2024-03-26T21:33:58.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4139999999999999e-06 2024-03-26T21:33:58.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335594+0000", 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4960000000000001e-06 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335607+0000", 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2513e-05 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335615+0000", 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8070000000000006e-06 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335683+0000", 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7102999999999998e-05 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:58.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335691+0000", 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5620000000000004e-06 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963592+0000", 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62790077300000002 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963694+0000", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010242399999999999 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963804+0000", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109621 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972506+0000", 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087021449999999997 2024-03-26T21:33:58.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972674+0000", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016797000000000001 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972747+0000", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3324999999999993e-05 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972795+0000", 2024-03-26T21:33:58.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7966000000000001e-05 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972830+0000", 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5083999999999998e-05 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197779 7.9 7:9aaf6e48:::benchmark_data_smithi055_81491_object78292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.213 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:33:58.214 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.7976552340000005, 2024-03-26T21:33:58.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62208113700000001, 2024-03-26T21:33:58.214 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.214 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.214 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197779 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353949+0000", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5600000000000001e-06 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353956+0000", 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1930000000000004e-06 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353971+0000", 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4969e-05 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353981+0000", 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2650000000000002e-06 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.354171+0000", 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019027399999999999 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.354181+0000", 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9229999999999997e-06 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966463+0000", 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.61228198 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966603+0000", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013977299999999999 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966724+0000", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121178 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972643+0000", 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.005919059 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972714+0000", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0844000000000001e-05 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.975915+0000", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032011370000000002 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.975977+0000", 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1993000000000001e-05 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976029+0000", 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1989e-05 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197785 7.9 7:97132301:::benchmark_data_smithi055_81491_object78298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.7942994150000002, 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62510192499999995, 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.221 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.222 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.222 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.222 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197785 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:33:58.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357305+0000", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.657e-06 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357312+0000", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5239999999999997e-06 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357328+0000", 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6161e-05 2024-03-26T21:33:58.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357338+0000", 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6849999999999996e-06 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:58.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:58.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357447+0000", 2024-03-26T21:33:58.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109069 2024-03-26T21:33:58.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357457+0000", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0193e-05 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969600+0000", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.612143192 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969722+0000", 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121581 2024-03-26T21:33:58.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969844+0000", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000122569 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972685+0000", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002840609 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972864+0000", 2024-03-26T21:33:58.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017964700000000001 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982308+0000", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0094434640000000004 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982361+0000", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2800000000000003e-05 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982405+0000", 2024-03-26T21:33:58.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4774000000000002e-05 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197787 7.9 7:92feca94:::benchmark_data_smithi055_81491_object78300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.7925306890000003, 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62348025900000004, 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.230 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197787 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359073+0000", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.094e-06 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359079+0000", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1209999999999998e-06 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359090+0000", 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1576000000000001e-05 2024-03-26T21:33:58.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359098+0000", 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1599999999999998e-06 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359158+0000", 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9351999999999997e-05 2024-03-26T21:33:58.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359166+0000", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9670000000000001e-06 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972892+0000", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.61372603000000003 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972997+0000", 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105892 2024-03-26T21:33:58.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.973118+0000", 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012077400000000001 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982176+0000", 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0090576570000000002 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982219+0000", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3080000000000001e-05 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982465+0000", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00024610700000000002 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982514+0000", 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9280999999999998e-05 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982553+0000", 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8167999999999998e-05 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197819 7.9 7:958a9edc:::benchmark_data_smithi055_81491_object78332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.761357855, 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.59241795100000005, 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.237 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197819 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.238 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390247+0000", 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.401e-06 2024-03-26T21:33:58.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390253+0000", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2020000000000001e-06 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390266+0000", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2906e-05 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390272+0000", 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1279999999999996e-06 2024-03-26T21:33:58.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390325+0000", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2904e-05 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390335+0000", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0115000000000001e-05 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976112+0000", 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.58577727000000002 2024-03-26T21:33:58.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976265+0000", 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000153177 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976371+0000", 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105565 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982232+0000", 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058610119999999996 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982268+0000", 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5797999999999998e-05 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982592+0000", 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032408799999999999 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982631+0000", 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9313000000000001e-05 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982663+0000", 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2072000000000001e-05 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197831 7.9 7:9fc20ae5:::benchmark_data_smithi055_81491_object78344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.244 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.7513768089999999, 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.59832579299999999, 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197831 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:33:58.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400227+0000", 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0500000000000002e-07 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400231+0000", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0069999999999999e-06 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400241+0000", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0274e-05 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400249+0000", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8520000000000004e-06 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400413+0000", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000163886 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400421+0000", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9389999999999997e-06 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979179+0000", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.57875759599999999 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979291+0000", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112336 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979406+0000", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011535300000000001 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989307+0000", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0099010869999999994 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992452+0000", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0031449859999999998 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998396+0000", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0059438269999999996 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998492+0000", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6281999999999997e-05 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998552+0000", 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9463000000000003e-05 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197855 7.9 7:9ae59bd5:::benchmark_data_smithi055_81491_object78368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.7208808060000003, 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.57252650800000005, 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.251 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197855 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430723+0000", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7400000000000002e-07 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430728+0000", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9740000000000001e-06 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430736+0000", 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0020000000000006e-06 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430743+0000", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7780000000000002e-06 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430814+0000", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0998e-05 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430822+0000", 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5890000000000003e-06 2024-03-26T21:33:58.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.985967+0000", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.55514456499999998 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.986084+0000", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117447 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.986200+0000", 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011593400000000001 2024-03-26T21:33:58.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992507+0000", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0063063190000000003 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992520+0000", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.346e-05 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003156+0000", 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010636174 2024-03-26T21:33:58.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003202+0000", 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6131999999999998e-05 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003249+0000", 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6261999999999999e-05 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197875 7.9 7:90cfef7a:::benchmark_data_smithi055_81491_object78388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.6988129269999996, 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.55058782699999997, 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197875 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.258 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452792+0000", 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.485e-06 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452797+0000", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9699999999999998e-06 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452810+0000", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3777999999999999e-05 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452820+0000", 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4769999999999992e-06 2024-03-26T21:33:58.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452882+0000", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1804000000000007e-05 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452891+0000", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.098e-06 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989405+0000", 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.536514355 2024-03-26T21:33:58.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989510+0000", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104522 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989625+0000", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115179 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992561+0000", 2024-03-26T21:33:58.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029358219999999998 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998566+0000", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0060056479999999997 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003296+0000", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0047299539999999998 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003340+0000", 2024-03-26T21:33:58.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3852e-05 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003378+0000", 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7883000000000003e-05 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197878 7.9 7:990f5c53:::benchmark_data_smithi055_81491_object78391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.6972996900000004, 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.54920376000000004, 2024-03-26T21:33:58.264 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197878 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.265 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454304+0000", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.048e-06 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454309+0000", 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9910000000000002e-06 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454320+0000", 2024-03-26T21:33:58.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1080000000000001e-05 2024-03-26T21:33:58.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454328+0000", 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6960000000000005e-06 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454387+0000", 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8471000000000003e-05 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.269 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454396+0000", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0589999999999992e-06 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992599+0000", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53820327800000001 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992702+0000", 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103361 2024-03-26T21:33:58.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992811+0000", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000108515 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998620+0000", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058094109999999996 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998674+0000", 2024-03-26T21:33:58.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3369999999999999e-05 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003426+0000", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004752399 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003468+0000", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2292000000000001e-05 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003507+0000", 2024-03-26T21:33:58.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8788999999999997e-05 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197883 7.9 7:9dafc3e4:::benchmark_data_smithi055_81491_object78396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.6902207919999999, 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.54529854600000005, 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.273 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197883 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461383+0000", 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4899999999999995e-07 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461386+0000", 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3869999999999999e-06 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461394+0000", 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3660000000000003e-06 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461400+0000", 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4749999999999998e-06 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461457+0000", 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7148000000000002e-05 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461466+0000", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2320000000000007e-06 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995439+0000", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53397292699999999 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995539+0000", 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9877000000000006e-05 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995657+0000", 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117469 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998662+0000", 2024-03-26T21:33:58.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030051320000000002 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003259+0000", 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004597584 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006449+0000", 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003189103 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006631+0000", 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018235899999999999 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006681+0000", 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9838e-05 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197887 7.9 7:976766be:::benchmark_data_smithi055_81491_object78400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.6840530410000003, 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53942058100000001, 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197887 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467551+0000", 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3399999999999999e-07 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467554+0000", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4470000000000002e-06 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467562+0000", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.909e-06 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467567+0000", 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9840000000000001e-06 2024-03-26T21:33:58.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467609+0000", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1995e-05 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467616+0000", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7499999999999997e-06 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998701+0000", 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53108542199999997 2024-03-26T21:33:58.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998812+0000", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111204 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.000049+0000", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0012364100000000001 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003387+0000", 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033380749999999998 2024-03-26T21:33:58.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006723+0000", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033363199999999998 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006761+0000", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8281999999999997e-05 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006926+0000", 2024-03-26T21:33:58.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016495 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006971+0000", 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4199000000000002e-05 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.287 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.290 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197891 7.9 7:92702fe6:::benchmark_data_smithi055_81491_object78404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 7.6733765930000004, 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53345969100000001, 2024-03-26T21:33:58.291 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197891 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:33:58.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478227+0000", 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7800000000000001e-07 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478231+0000", 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9500000000000003e-06 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478241+0000", 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8020000000000003e-06 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478247+0000", 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0469999999999999e-06 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478287+0000", 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0296000000000002e-05 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478292+0000", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2070000000000003e-06 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003550+0000", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.52525799399999995 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003672+0000", 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121983 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003796+0000", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012325500000000001 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.009910+0000", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0061143020000000003 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011217+0000", 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001307374 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011594+0000", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000376277 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011644+0000", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9979999999999999e-05 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011686+0000", 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2546000000000002e-05 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.298 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.299 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:33:58.299 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:33:58.299 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:33:58.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:58.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:58.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:33:58.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:33:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.630 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:58.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:33:58.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:33:58.758 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:33:58.772 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:33:58.788 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:33:58.794 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:33:58.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:58.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:58.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:58.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:59.060 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:33:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:33:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:33:59.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:33:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:33:59.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:59.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:59 smithi055 ceph-mon[29406]: pgmap v219: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6.7 MiB/s, 107 objects/s recovering 2024-03-26T21:33:59.319 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:33:59 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2328154208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:59.319 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:59 smithi055 ceph-mon[40587]: pgmap v219: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6.7 MiB/s, 107 objects/s recovering 2024-03-26T21:33:59.319 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:33:59 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2328154208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:59.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:33:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:33:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:33:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:59 smithi067 ceph-mon[33346]: pgmap v219: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6.7 MiB/s, 107 objects/s recovering 2024-03-26T21:33:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:33:59 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2328154208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:33:59.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:59.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:59.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:33:59.709 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:33:59.811 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:33:59.852 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:33:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:33:59.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:33:59.885 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:33:59.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:00.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:00.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:00.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:00.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:00.276 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:00.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:00.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:00.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:00.471 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:00.471 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[40587]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:34:00.471 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:00.471 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:00.471 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:00 smithi055 ceph-mon[29406]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:34:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:00 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:00 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:00 smithi067 ceph-mon[33346]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T21:34:00.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:00.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:34:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:00.642 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:00.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:00.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:00.837 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:00.939 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:00.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:00.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:01.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:01.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:01.344 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[40587]: pgmap v221: 148 pgs: 148 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 780 op/s; 1.9 MiB/s, 30 objects/s recovering 2024-03-26T21:34:01.344 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:01.344 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:01.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[29406]: pgmap v221: 148 pgs: 148 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 780 op/s; 1.9 MiB/s, 30 objects/s recovering 2024-03-26T21:34:01.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:01.345 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:01.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.391 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:01.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:01 smithi067 ceph-mon[33346]: pgmap v221: 148 pgs: 148 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 798 B/s rd, 0 B/s wr, 780 op/s; 1.9 MiB/s, 30 objects/s recovering 2024-03-26T21:34:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:01.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:01.570 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:01.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:01.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:01.783 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:01.885 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:01.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:34:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:01.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:01.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:02.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.088 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:02.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:02.237 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:02.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:02.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:02.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:02.530 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:02.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.671 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:02.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:02.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:02.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:02 smithi055 ceph-mon[40587]: pgmap v222: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 838 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T21:34:02.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:02 smithi055 ceph-mon[40587]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:34:02.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:02 smithi055 ceph-mon[29406]: pgmap v222: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 838 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T21:34:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:02 smithi055 ceph-mon[29406]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:34:02.916 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:02.958 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:02.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:02 smithi067 ceph-mon[33346]: pgmap v222: 148 pgs: 148 active+clean; 2.1 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 838 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T21:34:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:02 smithi067 ceph-mon[33346]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T21:34:03.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.089 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:03.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:03.191 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:03.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:03.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:03.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:03.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:03.535 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:03.637 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.683 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:03.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:03.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:03.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:03.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:03.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:04.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:04.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:04.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:04.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:04.435 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:04.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:04.457 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:04.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:04.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:04.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T21:34:04.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.785 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:04.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:04.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:04.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:04.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:04 smithi067 ceph-mon[33346]: pgmap v224: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1484524589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:04 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:05.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.059 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:05.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:05.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[29406]: pgmap v224: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1484524589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:05.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[40587]: pgmap v224: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1484524589' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:04 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:34:05.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:05.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:05.304 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:05.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:05.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:05.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:05.614 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:05.642 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:34:05.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:05.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:05.884 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:05.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:05.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:05 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:34:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:05 smithi067 ceph-mon[33346]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:34:05.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:05 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3396427035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:05.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:05 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:06.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[40587]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3396427035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[29406]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3396427035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:06.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:05 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:34:06.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:06.144 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:06.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:06.283 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:06.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:06.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:06.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:06.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:06.669 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:06.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:06.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:06.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:06.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:06.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:06 smithi067 ceph-mon[33346]: pgmap v226: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 847 op/s 2024-03-26T21:34:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:06 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:06 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:34:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:06 smithi067 ceph-mon[33346]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:34:07.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[40587]: pgmap v226: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 847 op/s 2024-03-26T21:34:07.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:07.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:34:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[40587]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:34:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[29406]: pgmap v226: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 847 op/s 2024-03-26T21:34:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:34:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:06 smithi055 ceph-mon[29406]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T21:34:07.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:07.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:07.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:07.251 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:07.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:07.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.522 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:07.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:07.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:07.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:07.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:07.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:07.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:07.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:07 smithi067 ceph-mon[33346]: 7.a scrub starts 2024-03-26T21:34:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:07 smithi067 ceph-mon[33346]: 7.a scrub ok 2024-03-26T21:34:08.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:08.064 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[29406]: 7.a scrub starts 2024-03-26T21:34:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[29406]: 7.a scrub ok 2024-03-26T21:34:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[40587]: 7.a scrub starts 2024-03-26T21:34:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:08.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:07 smithi055 ceph-mon[40587]: 7.a scrub ok 2024-03-26T21:34:08.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:08.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:08.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:08.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:08.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:08.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:08.498 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T21:34:08.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:08.603 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:08.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:08.733 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:08.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:34:08.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: pgmap v228: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: pgmap v228: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:34:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:34:08.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:08.879 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:08.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:08.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: pgmap v228: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:34:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:08 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:34:09.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:09.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:09.174 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:09.252 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:09.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:09.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:09.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:09.558 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:09.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:09.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:34:09.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.773 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:09.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:34:09.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:34:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:34:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:34:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:09 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:34:09.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:09.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:09.979 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:34:09.979 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:34:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:09 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:34:10.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:10.009 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:34:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:10.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:10.063 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:34:10.112 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:34:10.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:10.160 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:34:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.199 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:10.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:10.203 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:34:10.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:34:10.281 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:10.290 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:34:10.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:10.342 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:34:10.349 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:10.352 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:34:10.352 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:34:10.352 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:34:10.353 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:34:10.353 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:34:10.353 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:34:10.353 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:34:10.353 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:34:10.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:10.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:10.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:10.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:10.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:10.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:10.954 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:10.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 4.1d scrub starts 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 4.1d scrub ok 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: pgmap v229: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 989 op/s 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:34:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:34:10.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:34:10.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:34:10.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:34:10.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/731047020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:10.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:10 smithi067 ceph-mon[33346]: 7.7 scrub starts 2024-03-26T21:34:11.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:11.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 4.1d scrub starts 2024-03-26T21:34:11.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 4.1d scrub ok 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: pgmap v229: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 989 op/s 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/731047020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[29406]: 7.7 scrub starts 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 4.1d scrub starts 2024-03-26T21:34:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 4.1d scrub ok 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: pgmap v229: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 989 op/s 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/731047020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:11.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:10 smithi055 ceph-mon[40587]: 7.7 scrub starts 2024-03-26T21:34:11.084 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:11.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.219 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:11.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:11.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:11.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:11.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:11.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:11.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:11.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:11.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:11.776 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:11.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:11.907 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:11.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 6.6 scrub starts 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 6.6 scrub ok 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:34:11.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:34:11.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: 7.7 scrub ok 2024-03-26T21:34:11.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:11 smithi067 ceph-mon[33346]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:34:12.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:12.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:12.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 6.6 scrub starts 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 6.6 scrub ok 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: 7.7 scrub ok 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[29406]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 6.6 scrub starts 2024-03-26T21:34:12.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 6.6 scrub ok 2024-03-26T21:34:12.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:34:12.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:34:12.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: 7.7 scrub ok 2024-03-26T21:34:12.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:11 smithi055 ceph-mon[40587]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T21:34:12.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:12.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:12.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:12.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:12.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:12.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:12.583 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:12.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:12.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[40587]: pgmap v231: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:34:12.801 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[29406]: pgmap v231: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-26T21:34:12.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:34:12.802 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:12 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:34:12.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:12.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:12.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:12.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:12 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:34:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:12 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:34:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:12 smithi067 ceph-mon[33346]: pgmap v231: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-26T21:34:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:12 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:34:12.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:12 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:34:13.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:13.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:13.145 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:13.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:13.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:13.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:13.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:13.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:13.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:13.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:13.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:13.846 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:13.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:13.977 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:13.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 7.5 scrub starts 2024-03-26T21:34:13.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:13 smithi067 ceph-mon[33346]: 7.5 scrub ok 2024-03-26T21:34:14.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 7.5 scrub starts 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[40587]: 7.5 scrub ok 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:34:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 7.5 scrub starts 2024-03-26T21:34:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:13 smithi055 ceph-mon[29406]: 7.5 scrub ok 2024-03-26T21:34:14.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:14.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:14.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:14.252 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:14.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:14.382 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:14.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:14.523 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:14.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:14.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:34:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:14.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:14.781 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:14.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:14.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:14.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:14 smithi067 ceph-mon[33346]: pgmap v233: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T21:34:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:14 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:34:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:14 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:34:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:14 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:34:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:14 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:34:15.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:15.057 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[40587]: pgmap v233: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T21:34:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:34:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:34:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[29406]: pgmap v233: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:34:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:14 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:34:15.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:15.184 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.211 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:15.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:15.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:34:15.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:15.364 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:15.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:15.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:15.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:15.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:15.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:15.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:34:15.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:34:15.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:15 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:34:15.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:15.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:15.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:34:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:34:15.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:15 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:34:16.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:16.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:16.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:16.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:16.336 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:16.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:16.439 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:16.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:16.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:16.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:16.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:16.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:16.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:16.894 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:16.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:16.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: pgmap v234: 148 pgs: 148 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:34:16.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:16 smithi067 ceph-mon[33346]: 7.c scrub starts 2024-03-26T21:34:16.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:17.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:17.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: pgmap v234: 148 pgs: 148 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[29406]: 7.c scrub starts 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: pgmap v234: 148 pgs: 148 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 649 op/s 2024-03-26T21:34:17.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:34:17.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:34:17.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:17.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:34:17.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:34:17.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:16 smithi055 ceph-mon[40587]: 7.c scrub starts 2024-03-26T21:34:17.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:17.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:17.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:17.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:17.448 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:17.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:17.581 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:17.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:17.721 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:17.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:17.873 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[40587]: 7.c scrub ok 2024-03-26T21:34:17.873 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:34:17.873 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:34:17.873 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:17.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[29406]: 7.c scrub ok 2024-03-26T21:34:17.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:34:17.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:34:17.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:17.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:17.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:17 smithi067 ceph-mon[33346]: 7.c scrub ok 2024-03-26T21:34:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:17 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:34:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:17 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:34:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:18.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:18.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:18.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:18.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:18.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:18.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:18.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:18.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.647 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:18.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:18.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:18.771 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T21:34:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:18.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:18.923 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:18.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 965 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 7.3 scrub starts 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 7.8 scrub starts 2024-03-26T21:34:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 7.3 scrub ok 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[40587]: 7.8 scrub ok 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 965 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 7.3 scrub starts 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 7.8 scrub starts 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 7.3 scrub ok 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:34:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-mon[29406]: 7.8 scrub ok 2024-03-26T21:34:19.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:19.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:19.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.186 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: pgmap v235: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 965 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 7.3 scrub starts 2024-03-26T21:34:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:34:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:34:19.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 7.8 scrub starts 2024-03-26T21:34:19.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 7.3 scrub ok 2024-03-26T21:34:19.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:34:19.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:18 smithi067 ceph-mon[33346]: 7.8 scrub ok 2024-03-26T21:34:19.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:19.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:19.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.461 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:19.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:19.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.592 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:19.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:19.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:19.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:19.984 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:19.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:20.020 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:34:20.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:34:20.022 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:34:20.022 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:34:20.022 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:19 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:34:20.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:20.114 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:20.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:20.235 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:34:20.235 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:34:20.235 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:34:20.235 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:34:20.235 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3320509119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:34:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:19 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:34:20.260 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:20.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:20.404 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:20.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:20.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:20.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:20.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:20.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:20.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:20.962 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:20.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:21.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: pgmap v237: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 944 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:34:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: 7.b scrub starts 2024-03-26T21:34:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: 7.b scrub ok 2024-03-26T21:34:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:34:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:34:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: 7.d scrub starts 2024-03-26T21:34:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: pgmap v237: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 944 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: 7.b scrub starts 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: 7.b scrub ok 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: 7.d scrub starts 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:20 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:21.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:21.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:21.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:21.239 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: pgmap v237: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 944 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: 7.b scrub starts 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: 7.b scrub ok 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: 7.d scrub starts 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:20 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:21.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.385 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:21.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:21.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:21.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:34:21.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T21:34:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:21.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:21.658 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:21.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:21.694 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:21.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:21.840 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:34:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:21.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:21.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:21.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 7.d scrub ok 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:34:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:34:21.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:34:21.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:34:21.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:22.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 7.d scrub ok 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:34:22.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 7.d scrub ok 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:34:22.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:34:22.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:34:22.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:34:22.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:34:22.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T21:34:22.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:22.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:22.136 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:22.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:22.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:22.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.395 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:22.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:22.496 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.532 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:22.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:22.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:22.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:22.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:22.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:22.908 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:22.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:22.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:23.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[40587]: pgmap v239: 148 pgs: 148 active+clean; 840 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-03-26T21:34:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[40587]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[29406]: pgmap v239: 148 pgs: 148 active+clean; 840 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:22 smithi055 ceph-mon[29406]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:34:23.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:23.206 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.234 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:23.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:22 smithi067 ceph-mon[33346]: pgmap v239: 148 pgs: 148 active+clean; 840 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-03-26T21:34:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:22 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:34:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:22 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:34:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/734999770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:22 smithi067 ceph-mon[33346]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T21:34:23.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.364 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:23.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:23.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:23.510 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:23.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:23.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:23.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:23.875 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:23.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:23.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:24.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:24.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:24.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:24.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:23 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:34:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:23 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:34:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:23 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:34:24.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:23 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:34:24.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:24.315 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:34:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:34:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:34:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:34:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:34:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:34:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:34:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:23 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:34:24.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:24.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:24.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:24.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:24.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:24.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:24.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:24.983 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:25.010 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:25.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:25.133 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:25.148 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[29406]: pgmap v241: 148 pgs: 148 active+clean; 644 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s 2024-03-26T21:34:25.148 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:34:25.148 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:34:25.149 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[40587]: pgmap v241: 148 pgs: 148 active+clean; 644 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s 2024-03-26T21:34:25.149 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:34:25.149 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:24 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:34:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.179 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:25.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:24 smithi067 ceph-mon[33346]: pgmap v241: 148 pgs: 148 active+clean; 644 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 856 op/s 2024-03-26T21:34:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:24 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:34:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:24 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:34:25.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.322 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:25.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:25.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:25.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:25.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:25.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:25.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:25.845 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:25.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:25.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:25.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 6.1b deep-scrub starts 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 6.1b deep-scrub ok 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 3.0 deep-scrub starts 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 3.0 deep-scrub ok 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 6.1b deep-scrub starts 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 6.1b deep-scrub ok 2024-03-26T21:34:26.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 3.0 deep-scrub starts 2024-03-26T21:34:26.013 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 3.0 deep-scrub ok 2024-03-26T21:34:26.013 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:34:26.013 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:34:26.013 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:25 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:34:26.036 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:34:26.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:34:26.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:26.131 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:34:26.179 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:34:26.217 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:34:26.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 6.1b deep-scrub starts 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 6.1b deep-scrub ok 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 3.0 deep-scrub starts 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 3.0 deep-scrub ok 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:34:26.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:25 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:34:26.256 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:34:26.264 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:26.304 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:34:26.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:34:26.359 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:34:26.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:26.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:26.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:34:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.528 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:26.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:26.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: pgmap v242: 148 pgs: 148 active+clean; 625 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 773 B/s rd, 0 B/s wr, 771 op/s 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:34:26.854 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3237039471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:26.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:26.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:26.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:34:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:34:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:34:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: pgmap v242: 148 pgs: 148 active+clean; 625 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 773 B/s rd, 0 B/s wr, 771 op/s 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:34:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:26 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3237039471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:27.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:27.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:27.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: pgmap v242: 148 pgs: 148 active+clean; 625 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 773 B/s rd, 0 B/s wr, 771 op/s 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:34:27.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:34:27.238 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:34:27.238 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:34:27.238 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:26 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3237039471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:27.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:27.319 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:27.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:27.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:27.586 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:27.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:27.730 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:27.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:27.873 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:27.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:27.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.005 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:34:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:28.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:27 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:28.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:28.145 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:34:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:34:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:34:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:27 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:28.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:28.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:28.409 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:28.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:28.540 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:28.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:28.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:28.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:28.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:29.043 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:29.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:29.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:29.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 4.14 deep-scrub starts 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 4.14 deep-scrub ok 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 5.18 scrub starts 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 5.18 scrub ok 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 4.1f deep-scrub starts 2024-03-26T21:34:29.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[29406]: 4.1f deep-scrub ok 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 4.14 deep-scrub starts 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 4.14 deep-scrub ok 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 5.18 scrub starts 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 5.18 scrub ok 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 4.1f deep-scrub starts 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-mon[40587]: 4.1f deep-scrub ok 2024-03-26T21:34:29.229 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 4.14 deep-scrub starts 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 4.14 deep-scrub ok 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 5.18 scrub starts 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 5.18 scrub ok 2024-03-26T21:34:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T21:34:29.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 4.1f deep-scrub starts 2024-03-26T21:34:29.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:28 smithi067 ceph-mon[33346]: 4.1f deep-scrub ok 2024-03-26T21:34:29.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:34:29.306 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:29.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:29.378 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:29.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:29.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:29.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:29.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:29.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:29.940 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:29.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:29.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:30.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:30.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:34:30.144 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:34:30.145 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:29 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:34:30.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:30.203 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:34:30.203 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:34:30.216 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:29 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:34:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:29 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:34:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:29 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:34:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:29 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:34:30.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:30.450 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:30.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:30.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:30.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:30.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:30.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:30.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:30.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:31.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:31.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:34:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:31.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[40587]: pgmap v245: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 313 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[40587]: 2.1 deep-scrub starts 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[40587]: 2.1 deep-scrub ok 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:31.237 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[29406]: pgmap v245: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 313 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T21:34:31.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[29406]: 2.1 deep-scrub starts 2024-03-26T21:34:31.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[29406]: 2.1 deep-scrub ok 2024-03-26T21:34:31.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:31.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:30 smithi067 ceph-mon[33346]: pgmap v245: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 313 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T21:34:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:30 smithi067 ceph-mon[33346]: 2.1 deep-scrub starts 2024-03-26T21:34:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:30 smithi067 ceph-mon[33346]: 2.1 deep-scrub ok 2024-03-26T21:34:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:30 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:31.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:31.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:31.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.444 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:31.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:31.601 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Removed 119484 objects 2024-03-26T21:34:31.601 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Clean up completed and total clean up time :127.878 2024-03-26T21:34:31.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:31.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:31.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:31.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:31.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:31.952 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:31.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:32.053 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.080 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:32.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:32.211 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:34:32.211 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 4.1a deep-scrub starts 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[29406]: 4.1a deep-scrub ok 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 4.1a deep-scrub starts 2024-03-26T21:34:32.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:31 smithi055 ceph-mon[40587]: 4.1a deep-scrub ok 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 4.1a deep-scrub starts 2024-03-26T21:34:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:31 smithi067 ceph-mon[33346]: 4.1a deep-scrub ok 2024-03-26T21:34:32.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:32.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:32.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.458 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:32.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:32.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:32.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:32.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:32.839 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:32.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:32.966 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:32.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:33.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:33.144 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[40587]: pgmap v246: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 215 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 950 op/s 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[29406]: pgmap v246: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 215 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 950 op/s 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:34:33.145 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:32 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:34:33.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:33.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:32 smithi067 ceph-mon[33346]: pgmap v246: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 215 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 950 op/s 2024-03-26T21:34:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:32 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:34:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:32 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:34:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:32 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:34:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:32 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:34:33.278 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-26T21:34:33.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-26T21:34:33.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:33.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:33.367 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:33.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:33.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:33.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:33.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:33.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:33.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:33.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.868 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:33.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:33.913 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T21:34:33.928 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:34:33.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:33.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:34.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.128 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:34.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:34.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 7.4 scrub starts 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 7.4 scrub ok 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:34:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 5.6 deep-scrub starts 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 5.6 deep-scrub ok 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 3.a deep-scrub starts 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: 3.a deep-scrub ok 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/560014705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:33 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.254 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 7.4 scrub starts 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 7.4 scrub ok 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 5.6 deep-scrub starts 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 5.6 deep-scrub ok 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 3.a deep-scrub starts 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: 3.a deep-scrub ok 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/560014705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 7.4 scrub starts 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 7.4 scrub ok 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 5.6 deep-scrub starts 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 5.6 deep-scrub ok 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 3.a deep-scrub starts 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: 3.a deep-scrub ok 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/560014705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:33 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:34:34.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:34.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:34.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:34.516 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:34.617 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:34.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:34.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:34.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:34.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:34.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:35.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:35.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:35.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:35.166 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: pgmap v247: 148 pgs: 148 active+clean; 20 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:34:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:34:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:34:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:34:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:34 smithi067 ceph-mon[33346]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:34:35.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:35.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:35.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: pgmap v247: 148 pgs: 148 active+clean; 20 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[40587]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:34:35.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: pgmap v247: 148 pgs: 148 active+clean; 20 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1908133791' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:34 smithi055 ceph-mon[29406]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T21:34:35.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:35.525 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:35.550 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:35.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:35.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.811 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:35.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:35.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:35.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:36.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.072 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:36.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:36.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:36.202 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:35 smithi067 ceph-mon[33346]: 7.9 scrub starts 2024-03-26T21:34:36.304 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:36.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:35 smithi055 ceph-mon[29406]: 7.9 scrub starts 2024-03-26T21:34:36.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:35 smithi055 ceph-mon[40587]: 7.9 scrub starts 2024-03-26T21:34:36.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:36.392 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:36.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:36.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:36.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:36.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:36.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:36.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:36.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:36.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:36.975 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:36.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:36.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:36 smithi067 ceph-mon[33346]: pgmap v249: 148 pgs: 148 active+clean; 5.9 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s 2024-03-26T21:34:36.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:37.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.105 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:37.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:37.206 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:37.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:37.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:36 smithi055 ceph-mon[40587]: pgmap v249: 148 pgs: 148 active+clean; 5.9 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s 2024-03-26T21:34:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:36 smithi055 ceph-mon[29406]: pgmap v249: 148 pgs: 148 active+clean; 5.9 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s 2024-03-26T21:34:37.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:37.360 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:37.400 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T21:34:37.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:34:37.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:37.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:37.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:37.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:37.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:37.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:37.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:37.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.000 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:38.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:38.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:38.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.130 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:38.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:38.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:38.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:38.551 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:38.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:38.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:38.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:38.903 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:38.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:39.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:39.030 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:39.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:38 smithi055 ceph-mon[29406]: pgmap v250: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 561 op/s 2024-03-26T21:34:39.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:38 smithi055 ceph-mon[40587]: pgmap v250: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 561 op/s 2024-03-26T21:34:39.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:39.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:38 smithi067 ceph-mon[33346]: pgmap v250: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 561 op/s 2024-03-26T21:34:39.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:39.289 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:39.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.418 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:39.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:39.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:39.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:39.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.673 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:39.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:39.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:39.851 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:39.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:39.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:39.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:40.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:40.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:40.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:40.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:40.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:40.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:40.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:40.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:40.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:34:40.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:40.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:40.695 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:40.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:40.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:40.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:40.949 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:40.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:41.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:41.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:41.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[29406]: pgmap v251: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s 2024-03-26T21:34:41.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:41.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:41.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[29406]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:34:41.161 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:41.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[40587]: pgmap v251: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s 2024-03-26T21:34:41.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:41.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:41.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[40587]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:34:41.162 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:40 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:41.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:41.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:40 smithi067 ceph-mon[33346]: pgmap v251: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 500 op/s 2024-03-26T21:34:41.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:40 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:41.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:40 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:41.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:40 smithi067 ceph-mon[33346]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T21:34:41.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:40 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:34:41.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:41.381 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:41.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:41.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:41.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:41.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:41.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:41.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:41.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:41.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:41.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:41.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:42.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:42.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.140 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:42.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:42.215 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:34:42.215 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:34:42.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:42.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:42.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[40587]: pgmap v253: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-26T21:34:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:42 smithi055 ceph-mon[29406]: pgmap v253: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-26T21:34:42.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:42.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:42 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:42 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:42.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:42 smithi067 ceph-mon[33346]: pgmap v253: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-26T21:34:42.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.516 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:42.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:42.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:42.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:42.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:42.743 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:42.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:42.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:42.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:42.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:34:42.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:43.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:43.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.145 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:43.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:43.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:43.372 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.397 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:43.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:43.498 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:43.522 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:43.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.649 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:43.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:43.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:43.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:43.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:43.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:43.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:44.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:44.024 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:44.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:44.154 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:44.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.281 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:44.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:44.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:44.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:44.424 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:34:44.445 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:34:44.467 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:34:44.487 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:34:44.511 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:34:44.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:44.535 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:34:44.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:44.557 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:34:44.577 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:34:44.579 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:34:44.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:44.642 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:44 smithi055 ceph-mon[29406]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:34:44.643 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3511555821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:44.643 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:44 smithi055 ceph-mon[40587]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:34:44.643 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3511555821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:44.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:44 smithi067 ceph-mon[33346]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T21:34:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3511555821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:44.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.796 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:44.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:44.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:44.920 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:44.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:45.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:45.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:45.078 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:45.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:45.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:45.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:45.336 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:34:45.338 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:34:45.410 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:45.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:45.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:45 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:45 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:45.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:45 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:45.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:45 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:45.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:45.690 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:45.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:45.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:45.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:45.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:45.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:45 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:45 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:46.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.070 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:46.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:46.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:46.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:46.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:46.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:46.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:46.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:46.712 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:46.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:46.841 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:46.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:46.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:46.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:46.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:34:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:46.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:47.013 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='client.24908 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: pgmap v255: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 9 op/s 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:47.014 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:34:47.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:34:47.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:34:47.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:34:47.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:34:47.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:34:47.039 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-03-26T21:34:43.392902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1287,"num_read_kb":1133,"num_write":239585,"num_write_kb":7649438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19792,"num_bytes_recovered":1297090502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90156,"ondisk_log_size":90156,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1152324,"kb_used_data":9220,"kb_used_omap":15,"kb_used_meta":1141552,"kb_avail":748808892,"statfs":{"total":767960285184,"available":766780305408,"internally_reserved":0,"allocated":9441280,"data_stored":5780671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1168950030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5570584,"num_objects":-86,"num_object_clones":0,"num_object_copies":-258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-86,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":86,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.465177"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":133,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920175+0000","last_change":"2024-03-26T21:32:55.466524+0000","last_active":"2024-03-26T21:34:33.920175+0000","last_peered":"2024-03-26T21:34:33.920175+0000","last_clean":"2024-03-26T21:34:33.920175+0000","last_became_active":"2024-03-26T21:32:55.466152+0000","last_became_peered":"2024-03-26T21:32:55.466152+0000","last_unstale":"2024-03-26T21:34:33.920175+0000","last_undegraded":"2024-03-26T21:34:33.920175+0000","last_fullsized":"2024-03-26T21:34:33.920175+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975295+0000","last_change":"2024-03-26T21:34:20.509053+0000","last_active":"2024-03-26T21:34:39.975295+0000","last_peered":"2024-03-26T21:34:39.975295+0000","last_clean":"2024-03-26T21:34:39.975295+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:34:39.975295+0000","last_undegraded":"2024-03-26T21:34:39.975295+0000","last_fullsized":"2024-03-26T21:34:39.975295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.508804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.508804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:37.144539+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974905+0000","last_change":"2024-03-26T21:32:14.926305+0000","last_active":"2024-03-26T21:34:39.974905+0000","last_peered":"2024-03-26T21:34:39.974905+0000","last_clean":"2024-03-26T21:34:39.974905+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:34:39.974905+0000","last_undegraded":"2024-03-26T21:34:39.974905+0000","last_fullsized":"2024-03-26T21:34:39.974905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:14.926185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:32:14.926185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":161,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920043+0000","last_change":"2024-03-26T21:34:20.720386+0000","last_active":"2024-03-26T21:34:33.920043+0000","last_peered":"2024-03-26T21:34:33.920043+0000","last_clean":"2024-03-26T21:34:33.920043+0000","last_became_active":"2024-03-26T21:32:55.468198+0000","last_became_peered":"2024-03-26T21:32:55.468198+0000","last_unstale":"2024-03-26T21:34:33.920043+0000","last_undegraded":"2024-03-26T21:34:33.920043+0000","last_fullsized":"2024-03-26T21:34:33.920043+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:20.720301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.720301+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:26.959221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078177+0000","last_change":"2024-03-26T21:34:24.368408+0000","last_active":"2024-03-26T21:34:40.078177+0000","last_peered":"2024-03-26T21:34:40.078177+0000","last_clean":"2024-03-26T21:34:40.078177+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:34:40.078177+0000","last_undegraded":"2024-03-26T21:34:40.078177+0000","last_fullsized":"2024-03-26T21:34:40.078177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.368161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.690462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":141,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.919891+0000","last_change":"2024-03-26T21:34:27.847069+0000","last_active":"2024-03-26T21:34:33.919891+0000","last_peered":"2024-03-26T21:34:33.919891+0000","last_clean":"2024-03-26T21:34:33.919891+0000","last_became_active":"2024-03-26T21:32:55.468736+0000","last_became_peered":"2024-03-26T21:32:55.468736+0000","last_unstale":"2024-03-26T21:34:33.919891+0000","last_undegraded":"2024-03-26T21:34:33.919891+0000","last_fullsized":"2024-03-26T21:34:33.919891+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.846942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.846942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.712678+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078187+0000","last_change":"2024-03-26T21:34:30.380605+0000","last_active":"2024-03-26T21:34:40.078187+0000","last_peered":"2024-03-26T21:34:40.078187+0000","last_clean":"2024-03-26T21:34:40.078187+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:34:40.078187+0000","last_undegraded":"2024-03-26T21:34:40.078187+0000","last_fullsized":"2024-03-26T21:34:40.078187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:30.379661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.379661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.318868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":254,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078343+0000","last_change":"2024-03-26T21:32:13.108197+0000","last_active":"2024-03-26T21:34:40.078343+0000","last_peered":"2024-03-26T21:34:40.078343+0000","last_clean":"2024-03-26T21:34:40.078343+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:34:40.078343+0000","last_undegraded":"2024-03-26T21:34:40.078343+0000","last_fullsized":"2024-03-26T21:34:40.078343+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:13.107771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.107771+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284398+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.973924+0000","last_change":"2024-03-26T21:31:42.540278+0000","last_active":"2024-03-26T21:34:39.973924+0000","last_peered":"2024-03-26T21:34:39.973924+0000","last_clean":"2024-03-26T21:34:39.973924+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:34:39.973924+0000","last_undegraded":"2024-03-26T21:34:39.973924+0000","last_fullsized":"2024-03-26T21:34:39.973924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:42.540161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:42.540161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016199+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079435+0000","last_change":"2024-03-26T21:34:31.792345+0000","last_active":"2024-03-26T21:34:40.079435+0000","last_peered":"2024-03-26T21:34:40.079435+0000","last_clean":"2024-03-26T21:34:40.079435+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:34:40.079435+0000","last_undegraded":"2024-03-26T21:34:40.079435+0000","last_fullsized":"2024-03-26T21:34:40.079435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:31.792268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.792268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.823508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078797+0000","last_change":"2024-03-26T21:33:22.738629+0000","last_active":"2024-03-26T21:34:40.078797+0000","last_peered":"2024-03-26T21:34:40.078797+0000","last_clean":"2024-03-26T21:34:40.078797+0000","last_became_active":"2024-03-26T21:32:55.465246+0000","last_became_peered":"2024-03-26T21:32:55.465246+0000","last_unstale":"2024-03-26T21:34:40.078797+0000","last_undegraded":"2024-03-26T21:34:40.078797+0000","last_fullsized":"2024-03-26T21:34:40.078797+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.738510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.738510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"167'13","reported_seq":165,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920075+0000","last_change":"2024-03-26T21:34:30.923713+0000","last_active":"2024-03-26T21:34:33.920075+0000","last_peered":"2024-03-26T21:34:33.920075+0000","last_clean":"2024-03-26T21:34:33.920075+0000","last_became_active":"2024-03-26T21:32:55.480936+0000","last_became_peered":"2024-03-26T21:32:55.480936+0000","last_unstale":"2024-03-26T21:34:33.920075+0000","last_undegraded":"2024-03-26T21:34:33.920075+0000","last_fullsized":"2024-03-26T21:34:33.920075+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.923562+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:40.154880+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077566+0000","last_change":"2024-03-26T21:34:22.762001+0000","last_active":"2024-03-26T21:34:40.077566+0000","last_peered":"2024-03-26T21:34:40.077566+0000","last_clean":"2024-03-26T21:34:40.077566+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:34:40.077566+0000","last_undegraded":"2024-03-26T21:34:40.077566+0000","last_fullsized":"2024-03-26T21:34:40.077566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:22.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:34:22.761871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.252793+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":121,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077760+0000","last_change":"2024-03-26T21:34:23.724557+0000","last_active":"2024-03-26T21:34:40.077760+0000","last_peered":"2024-03-26T21:34:40.077760+0000","last_clean":"2024-03-26T21:34:40.077760+0000","last_became_active":"2024-03-26T21:32:20.904068+0000","last_became_peered":"2024-03-26T21:32:20.904068+0000","last_unstale":"2024-03-26T21:34:40.077760+0000","last_undegraded":"2024-03-26T21:34:40.077760+0000","last_fullsized":"2024-03-26T21:34:40.077760+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.724419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.724419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.040112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078348+0000","last_change":"2024-03-26T21:33:25.751247+0000","last_active":"2024-03-26T21:34:40.078348+0000","last_peered":"2024-03-26T21:34:40.078348+0000","last_clean":"2024-03-26T21:34:40.078348+0000","last_became_active":"2024-03-26T21:32:55.466491+0000","last_became_peered":"2024-03-26T21:32:55.466491+0000","last_unstale":"2024-03-26T21:34:40.078348+0000","last_undegraded":"2024-03-26T21:34:40.078348+0000","last_fullsized":"2024-03-26T21:34:40.078348+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:25.751149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:25.751149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":120,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920643+0000","last_change":"2024-03-26T21:34:11.780408+0000","last_active":"2024-03-26T21:34:33.920643+0000","last_peered":"2024-03-26T21:34:33.920643+0000","last_clean":"2024-03-26T21:34:33.920643+0000","last_became_active":"2024-03-26T21:32:55.476272+0000","last_became_peered":"2024-03-26T21:32:55.476272+0000","last_unstale":"2024-03-26T21:34:33.920643+0000","last_undegraded":"2024-03-26T21:34:33.920643+0000","last_fullsized":"2024-03-26T21:34:33.920643+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:34:11.780127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.780127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:13.819455+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":98,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920991+0000","last_change":"2024-03-26T21:34:06.784378+0000","last_active":"2024-03-26T21:34:33.920991+0000","last_peered":"2024-03-26T21:34:33.920991+0000","last_clean":"2024-03-26T21:34:33.920991+0000","last_became_active":"2024-03-26T21:32:55.467335+0000","last_became_peered":"2024-03-26T21:32:55.467335+0000","last_unstale":"2024-03-26T21:34:33.920991+0000","last_undegraded":"2024-03-26T21:34:33.920991+0000","last_fullsized":"2024-03-26T21:34:33.920991+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:06.784228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:06.784228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:18.860098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":234,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976402+0000","last_change":"2024-03-26T21:33:18.800182+0000","last_active":"2024-03-26T21:34:39.976402+0000","last_peered":"2024-03-26T21:34:39.976402+0000","last_clean":"2024-03-26T21:34:39.976402+0000","last_became_active":"2024-03-26T21:32:55.472908+0000","last_became_peered":"2024-03-26T21:32:55.472908+0000","last_unstale":"2024-03-26T21:34:39.976402+0000","last_undegraded":"2024-03-26T21:34:39.976402+0000","last_fullsized":"2024-03-26T21:34:39.976402+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:18.799990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:18.799990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.371148+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974751+0000","last_change":"2024-03-26T21:34:27.088958+0000","last_active":"2024-03-26T21:34:39.974751+0000","last_peered":"2024-03-26T21:34:39.974751+0000","last_clean":"2024-03-26T21:34:39.974751+0000","last_became_active":"2024-03-26T21:32:55.479242+0000","last_became_peered":"2024-03-26T21:32:55.479242+0000","last_unstale":"2024-03-26T21:34:39.974751+0000","last_undegraded":"2024-03-26T21:34:39.974751+0000","last_fullsized":"2024-03-26T21:34:39.974751+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.088840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.088840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:49.279253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":262,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077936+0000","last_change":"2024-03-26T21:32:29.620716+0000","last_active":"2024-03-26T21:34:40.077936+0000","last_peered":"2024-03-26T21:34:40.077936+0000","last_clean":"2024-03-26T21:34:40.077936+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:34:40.077936+0000","last_undegraded":"2024-03-26T21:34:40.077936+0000","last_fullsized":"2024-03-26T21:34:40.077936+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:29.620574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.620574+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.880673+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078731+0000","last_change":"2024-03-26T21:32:30.360160+0000","last_active":"2024-03-26T21:34:40.078731+0000","last_peered":"2024-03-26T21:34:40.078731+0000","last_clean":"2024-03-26T21:34:40.078731+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:34:40.078731+0000","last_undegraded":"2024-03-26T21:34:40.078731+0000","last_fullsized":"2024-03-26T21:34:40.078731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.360050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.360050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284416+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":141,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.921172+0000","last_change":"2024-03-26T21:34:25.862079+0000","last_active":"2024-03-26T21:34:33.921172+0000","last_peered":"2024-03-26T21:34:33.921172+0000","last_clean":"2024-03-26T21:34:33.921172+0000","last_became_active":"2024-03-26T21:32:55.465947+0000","last_became_peered":"2024-03-26T21:32:55.465947+0000","last_unstale":"2024-03-26T21:34:33.921172+0000","last_undegraded":"2024-03-26T21:34:33.921172+0000","last_fullsized":"2024-03-26T21:34:33.921172+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.861995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.861995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.431656+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":145,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920404+0000","last_change":"2024-03-26T21:32:55.477901+0000","last_active":"2024-03-26T21:34:33.920404+0000","last_peered":"2024-03-26T21:34:33.920404+0000","last_clean":"2024-03-26T21:34:33.920404+0000","last_became_active":"2024-03-26T21:32:55.477250+0000","last_became_peered":"2024-03-26T21:32:55.477250+0000","last_unstale":"2024-03-26T21:34:33.920404+0000","last_undegraded":"2024-03-26T21:34:33.920404+0000","last_fullsized":"2024-03-26T21:34:33.920404+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":253,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077633+0000","last_change":"2024-03-26T21:34:09.286797+0000","last_active":"2024-03-26T21:34:40.077633+0000","last_peered":"2024-03-26T21:34:40.077633+0000","last_clean":"2024-03-26T21:34:40.077633+0000","last_became_active":"2024-03-26T21:32:55.470311+0000","last_became_peered":"2024-03-26T21:32:55.470311+0000","last_unstale":"2024-03-26T21:34:40.077633+0000","last_undegraded":"2024-03-26T21:34:40.077633+0000","last_fullsized":"2024-03-26T21:34:40.077633+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:09.286636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.286636+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.511715+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":238,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975751+0000","last_change":"2024-03-26T21:34:25.733995+0000","last_active":"2024-03-26T21:34:39.975751+0000","last_peered":"2024-03-26T21:34:39.975751+0000","last_clean":"2024-03-26T21:34:39.975751+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:34:39.975751+0000","last_undegraded":"2024-03-26T21:34:39.975751+0000","last_fullsized":"2024-03-26T21:34:39.975751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:34:25.733888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.733888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:27.393061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"134'10","reported_seq":329,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079575+0000","last_change":"2024-03-26T21:34:27.049262+0000","last_active":"2024-03-26T21:34:40.079575+0000","last_peered":"2024-03-26T21:34:40.079575+0000","last_clean":"2024-03-26T21:34:40.079575+0000","last_became_active":"2024-03-26T21:32:55.477677+0000","last_became_peered":"2024-03-26T21:32:55.477677+0000","last_unstale":"2024-03-26T21:34:40.079575+0000","last_undegraded":"2024-03-26T21:34:40.079575+0000","last_fullsized":"2024-03-26T21:34:40.079575+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:27.048989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.048989+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.859023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":244,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975391+0000","last_change":"2024-03-26T21:33:24.786987+0000","last_active":"2024-03-26T21:34:39.975391+0000","last_peered":"2024-03-26T21:34:39.975391+0000","last_clean":"2024-03-26T21:34:39.975391+0000","last_became_active":"2024-03-26T21:32:55.469277+0000","last_became_peered":"2024-03-26T21:32:55.469277+0000","last_unstale":"2024-03-26T21:34:39.975391+0000","last_undegraded":"2024-03-26T21:34:39.975391+0000","last_fullsized":"2024-03-26T21:34:39.975391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:33:24.786583+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:33:24.786583+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":245,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975987+0000","last_change":"2024-03-26T21:32:22.112186+0000","last_active":"2024-03-26T21:34:39.975987+0000","last_peered":"2024-03-26T21:34:39.975987+0000","last_clean":"2024-03-26T21:34:39.975987+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:34:39.975987+0000","last_undegraded":"2024-03-26T21:34:39.975987+0000","last_fullsized":"2024-03-26T21:34:39.975987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:32:22.112001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.112001+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:46.058568+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976249+0000","last_change":"2024-03-26T21:33:12.751050+0000","last_active":"2024-03-26T21:34:39.976249+0000","last_peered":"2024-03-26T21:34:39.976249+0000","last_clean":"2024-03-26T21:34:39.976249+0000","last_became_active":"2024-03-26T21:32:55.465456+0000","last_became_peered":"2024-03-26T21:32:55.465456+0000","last_unstale":"2024-03-26T21:34:39.976249+0000","last_undegraded":"2024-03-26T21:34:39.976249+0000","last_fullsized":"2024-03-26T21:34:39.976249+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:12.750915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:12.750915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.570116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":225,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976124+0000","last_change":"2024-03-26T21:31:28.707417+0000","last_active":"2024-03-26T21:34:39.976124+0000","last_peered":"2024-03-26T21:34:39.976124+0000","last_clean":"2024-03-26T21:34:39.976124+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:34:39.976124+0000","last_undegraded":"2024-03-26T21:34:39.976124+0000","last_fullsized":"2024-03-26T21:34:39.976124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.707018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016112+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077753+0000","last_change":"2024-03-26T21:31:39.883235+0000","last_active":"2024-03-26T21:34:40.077753+0000","last_peered":"2024-03-26T21:34:40.077753+0000","last_clean":"2024-03-26T21:34:40.077753+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:34:40.077753+0000","last_undegraded":"2024-03-26T21:34:40.077753+0000","last_fullsized":"2024-03-26T21:34:40.077753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.883105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.883105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284356+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"167'16","reported_seq":269,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079334+0000","last_change":"2024-03-26T21:34:28.014694+0000","last_active":"2024-03-26T21:34:40.079334+0000","last_peered":"2024-03-26T21:34:40.079334+0000","last_clean":"2024-03-26T21:34:40.079334+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:34:40.079334+0000","last_undegraded":"2024-03-26T21:34:40.079334+0000","last_fullsized":"2024-03-26T21:34:40.079334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'16","last_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.014533+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.217243+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975333+0000","last_change":"2024-03-26T21:33:22.621399+0000","last_active":"2024-03-26T21:34:39.975333+0000","last_peered":"2024-03-26T21:34:39.975333+0000","last_clean":"2024-03-26T21:34:39.975333+0000","last_became_active":"2024-03-26T21:32:55.471141+0000","last_became_peered":"2024-03-26T21:32:55.471141+0000","last_unstale":"2024-03-26T21:34:39.975333+0000","last_undegraded":"2024-03-26T21:34:39.975333+0000","last_fullsized":"2024-03-26T21:34:39.975333+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.621257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.621257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:48.031289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078300+0000","last_change":"2024-03-26T21:34:23.051204+0000","last_active":"2024-03-26T21:34:40.078300+0000","last_peered":"2024-03-26T21:34:40.078300+0000","last_clean":"2024-03-26T21:34:40.078300+0000","last_became_active":"2024-03-26T21:32:55.469690+0000","last_became_peered":"2024-03-26T21:32:55.469690+0000","last_unstale":"2024-03-26T21:34:40.078300+0000","last_undegraded":"2024-03-26T21:34:40.078300+0000","last_fullsized":"2024-03-26T21:34:40.078300+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.051104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.051104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.773972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975093+0000","last_change":"2024-03-26T21:32:36.887352+0000","last_active":"2024-03-26T21:34:39.975093+0000","last_peered":"2024-03-26T21:34:39.975093+0000","last_clean":"2024-03-26T21:34:39.975093+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:34:39.975093+0000","last_undegraded":"2024-03-26T21:34:39.975093+0000","last_fullsized":"2024-03-26T21:34:39.975093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:36.887160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.887160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016280+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":243,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077863+0000","last_change":"2024-03-26T21:34:25.343996+0000","last_active":"2024-03-26T21:34:40.077863+0000","last_peered":"2024-03-26T21:34:40.077863+0000","last_clean":"2024-03-26T21:34:40.077863+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:34:40.077863+0000","last_undegraded":"2024-03-26T21:34:40.077863+0000","last_fullsized":"2024-03-26T21:34:40.077863+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:34:25.343633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.343633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.287164+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078486+0000","last_change":"2024-03-26T21:32:29.393768+0000","last_active":"2024-03-26T21:34:40.078486+0000","last_peered":"2024-03-26T21:34:40.078486+0000","last_clean":"2024-03-26T21:34:40.078486+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:34:40.078486+0000","last_undegraded":"2024-03-26T21:34:40.078486+0000","last_fullsized":"2024-03-26T21:34:40.078486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:29.393601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.393601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284552+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":226,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975613+0000","last_change":"2024-03-26T21:33:13.577559+0000","last_active":"2024-03-26T21:34:39.975613+0000","last_peered":"2024-03-26T21:34:39.975613+0000","last_clean":"2024-03-26T21:34:39.975613+0000","last_became_active":"2024-03-26T21:32:55.474210+0000","last_became_peered":"2024-03-26T21:32:55.474210+0000","last_unstale":"2024-03-26T21:34:39.975613+0000","last_undegraded":"2024-03-26T21:34:39.975613+0000","last_fullsized":"2024-03-26T21:34:39.975613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:13.577485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:13.577485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.526131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078838+0000","last_change":"2024-03-26T21:33:30.764236+0000","last_active":"2024-03-26T21:34:40.078838+0000","last_peered":"2024-03-26T21:34:40.078838+0000","last_clean":"2024-03-26T21:34:40.078838+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:34:40.078838+0000","last_undegraded":"2024-03-26T21:34:40.078838+0000","last_fullsized":"2024-03-26T21:34:40.078838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.763924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.763924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.316480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":248,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078303+0000","last_change":"2024-03-26T21:32:15.138279+0000","last_active":"2024-03-26T21:34:40.078303+0000","last_peered":"2024-03-26T21:34:40.078303+0000","last_clean":"2024-03-26T21:34:40.078303+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:34:40.078303+0000","last_undegraded":"2024-03-26T21:34:40.078303+0000","last_fullsized":"2024-03-26T21:34:40.078303+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:32:15.138133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:15.138133+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284378+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.973736+0000","last_change":"2024-03-26T21:32:26.906415+0000","last_active":"2024-03-26T21:34:39.973736+0000","last_peered":"2024-03-26T21:34:39.973736+0000","last_clean":"2024-03-26T21:34:39.973736+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:34:39.973736+0000","last_undegraded":"2024-03-26T21:34:39.973736+0000","last_fullsized":"2024-03-26T21:34:39.973736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.906307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.906307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016296+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":225,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077982+0000","last_change":"2024-03-26T21:31:32.871165+0000","last_active":"2024-03-26T21:34:40.077982+0000","last_peered":"2024-03-26T21:34:40.077982+0000","last_clean":"2024-03-26T21:34:40.077982+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:34:40.077982+0000","last_undegraded":"2024-03-26T21:34:40.077982+0000","last_fullsized":"2024-03-26T21:34:40.077982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:32.871039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.871039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284339+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975868+0000","last_change":"2024-03-26T21:33:17.583457+0000","last_active":"2024-03-26T21:34:39.975868+0000","last_peered":"2024-03-26T21:34:39.975868+0000","last_clean":"2024-03-26T21:34:39.975868+0000","last_became_active":"2024-03-26T21:32:55.470351+0000","last_became_peered":"2024-03-26T21:32:55.470351+0000","last_unstale":"2024-03-26T21:34:39.975868+0000","last_undegraded":"2024-03-26T21:34:39.975868+0000","last_fullsized":"2024-03-26T21:34:39.975868+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:17.583338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:17.583338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.475422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":257,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975948+0000","last_change":"2024-03-26T21:34:13.533465+0000","last_active":"2024-03-26T21:34:39.975948+0000","last_peered":"2024-03-26T21:34:39.975948+0000","last_clean":"2024-03-26T21:34:39.975948+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:34:39.975948+0000","last_undegraded":"2024-03-26T21:34:39.975948+0000","last_fullsized":"2024-03-26T21:34:39.975948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:34:13.533314+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.533314+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.625607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077586+0000","last_change":"2024-03-26T21:31:38.898103+0000","last_active":"2024-03-26T21:34:40.077586+0000","last_peered":"2024-03-26T21:34:40.077586+0000","last_clean":"2024-03-26T21:34:40.077586+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:34:40.077586+0000","last_undegraded":"2024-03-26T21:34:40.077586+0000","last_fullsized":"2024-03-26T21:34:40.077586+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:38.897810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.897810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284296+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975786+0000","last_change":"2024-03-26T21:34:25.554360+0000","last_active":"2024-03-26T21:34:39.975786+0000","last_peered":"2024-03-26T21:34:39.975786+0000","last_clean":"2024-03-26T21:34:39.975786+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:34:39.975786+0000","last_undegraded":"2024-03-26T21:34:39.975786+0000","last_fullsized":"2024-03-26T21:34:39.975786+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.554230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.554230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.463566+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":148,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920711+0000","last_change":"2024-03-26T21:32:55.467765+0000","last_active":"2024-03-26T21:34:33.920711+0000","last_peered":"2024-03-26T21:34:33.920711+0000","last_clean":"2024-03-26T21:34:33.920711+0000","last_became_active":"2024-03-26T21:32:55.467375+0000","last_became_peered":"2024-03-26T21:32:55.467375+0000","last_unstale":"2024-03-26T21:34:33.920711+0000","last_undegraded":"2024-03-26T21:34:33.920711+0000","last_fullsized":"2024-03-26T21:34:33.920711+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:26.155650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:26.155650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.921077+0000","last_change":"2024-03-26T21:34:10.824961+0000","last_active":"2024-03-26T21:34:33.921077+0000","last_peered":"2024-03-26T21:34:33.921077+0000","last_clean":"2024-03-26T21:34:33.921077+0000","last_became_active":"2024-03-26T21:32:55.477977+0000","last_became_peered":"2024-03-26T21:32:55.477977+0000","last_unstale":"2024-03-26T21:34:33.921077+0000","last_undegraded":"2024-03-26T21:34:33.921077+0000","last_fullsized":"2024-03-26T21:34:33.921077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:34:10.824829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.824829+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.688410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":222,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974682+0000","last_change":"2024-03-26T21:31:40.605350+0000","last_active":"2024-03-26T21:34:39.974682+0000","last_peered":"2024-03-26T21:34:39.974682+0000","last_clean":"2024-03-26T21:34:39.974682+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:34:39.974682+0000","last_undegraded":"2024-03-26T21:34:39.974682+0000","last_fullsized":"2024-03-26T21:34:39.974682+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:40.605038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:40.605038+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016164+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":316,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078889+0000","last_change":"2024-03-26T21:34:11.674348+0000","last_active":"2024-03-26T21:34:40.078889+0000","last_peered":"2024-03-26T21:34:40.078889+0000","last_clean":"2024-03-26T21:34:40.078889+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:34:40.078889+0000","last_undegraded":"2024-03-26T21:34:40.078889+0000","last_fullsized":"2024-03-26T21:34:40.078889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:34:11.674203+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.674203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:32.191741+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975427+0000","last_change":"2024-03-26T21:33:15.534591+0000","last_active":"2024-03-26T21:34:39.975427+0000","last_peered":"2024-03-26T21:34:39.975427+0000","last_clean":"2024-03-26T21:34:39.975427+0000","last_became_active":"2024-03-26T21:32:55.473442+0000","last_became_peered":"2024-03-26T21:32:55.473442+0000","last_unstale":"2024-03-26T21:34:39.975427+0000","last_undegraded":"2024-03-26T21:34:39.975427+0000","last_fullsized":"2024-03-26T21:34:39.975427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:15.534472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:15.534472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.526179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"166'15","reported_seq":257,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078033+0000","last_change":"2024-03-26T21:34:27.404044+0000","last_active":"2024-03-26T21:34:40.078033+0000","last_peered":"2024-03-26T21:34:40.078033+0000","last_clean":"2024-03-26T21:34:40.078033+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:34:40.078033+0000","last_undegraded":"2024-03-26T21:34:40.078033+0000","last_fullsized":"2024-03-26T21:34:40.078033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.403785+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.739092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920877+0000","last_change":"2024-03-26T21:32:55.466193+0000","last_active":"2024-03-26T21:34:33.920877+0000","last_peered":"2024-03-26T21:34:33.920877+0000","last_clean":"2024-03-26T21:34:33.920877+0000","last_became_active":"2024-03-26T21:32:55.465075+0000","last_became_peered":"2024-03-26T21:32:55.465075+0000","last_unstale":"2024-03-26T21:34:33.920877+0000","last_undegraded":"2024-03-26T21:34:33.920877+0000","last_fullsized":"2024-03-26T21:34:33.920877+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":310,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078224+0000","last_change":"2024-03-26T21:32:36.313231+0000","last_active":"2024-03-26T21:34:40.078224+0000","last_peered":"2024-03-26T21:34:40.078224+0000","last_clean":"2024-03-26T21:34:40.078224+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:34:40.078224+0000","last_undegraded":"2024-03-26T21:34:40.078224+0000","last_fullsized":"2024-03-26T21:34:40.078224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:36.311788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.311788+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284513+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974040+0000","last_change":"2024-03-26T21:32:23.910111+0000","last_active":"2024-03-26T21:34:39.974040+0000","last_peered":"2024-03-26T21:34:39.974040+0000","last_clean":"2024-03-26T21:34:39.974040+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:34:39.974040+0000","last_undegraded":"2024-03-26T21:34:39.974040+0000","last_fullsized":"2024-03-26T21:34:39.974040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.909981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.909981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"146'13","reported_seq":257,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079175+0000","last_change":"2024-03-26T21:33:29.776490+0000","last_active":"2024-03-26T21:34:40.079175+0000","last_peered":"2024-03-26T21:34:40.079175+0000","last_clean":"2024-03-26T21:34:40.079175+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:34:40.079175+0000","last_undegraded":"2024-03-26T21:34:40.079175+0000","last_fullsized":"2024-03-26T21:34:40.079175+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.776392+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:47.399458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078209+0000","last_change":"2024-03-26T21:34:14.715929+0000","last_active":"2024-03-26T21:34:40.078209+0000","last_peered":"2024-03-26T21:34:40.078209+0000","last_clean":"2024-03-26T21:34:40.078209+0000","last_became_active":"2024-03-26T21:32:55.468099+0000","last_became_peered":"2024-03-26T21:32:55.468099+0000","last_unstale":"2024-03-26T21:34:40.078209+0000","last_undegraded":"2024-03-26T21:34:40.078209+0000","last_fullsized":"2024-03-26T21:34:40.078209+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.715803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.715803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.624014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":225,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078949+0000","last_change":"2024-03-26T21:32:23.286349+0000","last_active":"2024-03-26T21:34:40.078949+0000","last_peered":"2024-03-26T21:34:40.078949+0000","last_clean":"2024-03-26T21:34:40.078949+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:34:40.078949+0000","last_undegraded":"2024-03-26T21:34:40.078949+0000","last_fullsized":"2024-03-26T21:34:40.078949+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.286238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.286238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284474+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975372+0000","last_change":"2024-03-26T21:33:23.637713+0000","last_active":"2024-03-26T21:34:39.975372+0000","last_peered":"2024-03-26T21:34:39.975372+0000","last_clean":"2024-03-26T21:34:39.975372+0000","last_became_active":"2024-03-26T21:32:55.470296+0000","last_became_peered":"2024-03-26T21:32:55.470296+0000","last_unstale":"2024-03-26T21:34:39.975372+0000","last_undegraded":"2024-03-26T21:34:39.975372+0000","last_fullsized":"2024-03-26T21:34:39.975372+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:23.637591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:23.637591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.475400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":249,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976055+0000","last_change":"2024-03-26T21:31:38.685593+0000","last_active":"2024-03-26T21:34:39.976055+0000","last_peered":"2024-03-26T21:34:39.976055+0000","last_clean":"2024-03-26T21:34:39.976055+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:34:39.976055+0000","last_undegraded":"2024-03-26T21:34:39.976055+0000","last_fullsized":"2024-03-26T21:34:39.976055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.685453+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016072+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975742+0000","last_change":"2024-03-26T21:33:27.579541+0000","last_active":"2024-03-26T21:34:39.975742+0000","last_peered":"2024-03-26T21:34:39.975742+0000","last_clean":"2024-03-26T21:34:39.975742+0000","last_became_active":"2024-03-26T21:32:55.470290+0000","last_became_peered":"2024-03-26T21:32:55.470290+0000","last_unstale":"2024-03-26T21:34:39.975742+0000","last_undegraded":"2024-03-26T21:34:39.975742+0000","last_fullsized":"2024-03-26T21:34:39.975742+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:27.579465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:33:27.579465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.584258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079167+0000","last_change":"2024-03-26T21:32:41.663300+0000","last_active":"2024-03-26T21:34:40.079167+0000","last_peered":"2024-03-26T21:34:40.079167+0000","last_clean":"2024-03-26T21:34:40.079167+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:34:40.079167+0000","last_undegraded":"2024-03-26T21:34:40.079167+0000","last_fullsized":"2024-03-26T21:34:40.079167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:41.663183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:41.663183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790373+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077362+0000","last_change":"2024-03-26T21:33:30.266914+0000","last_active":"2024-03-26T21:34:40.077362+0000","last_peered":"2024-03-26T21:34:40.077362+0000","last_clean":"2024-03-26T21:34:40.077362+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:34:40.077362+0000","last_undegraded":"2024-03-26T21:34:40.077362+0000","last_fullsized":"2024-03-26T21:34:40.077362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.266833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.266833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.214452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":252,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078885+0000","last_change":"2024-03-26T21:32:18.220547+0000","last_active":"2024-03-26T21:34:40.078885+0000","last_peered":"2024-03-26T21:34:40.078885+0000","last_clean":"2024-03-26T21:34:40.078885+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:34:40.078885+0000","last_undegraded":"2024-03-26T21:34:40.078885+0000","last_fullsized":"2024-03-26T21:34:40.078885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:32:18.214216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.214216+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284493+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"167'14780","reported_seq":15033,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975168+0000","last_change":"2024-03-26T21:34:19.726795+0000","last_active":"2024-03-26T21:34:39.975168+0000","last_peered":"2024-03-26T21:34:39.975168+0000","last_clean":"2024-03-26T21:34:39.975168+0000","last_became_active":"2024-03-26T21:30:38.543645+0000","last_became_peered":"2024-03-26T21:30:38.543645+0000","last_unstale":"2024-03-26T21:34:39.975168+0000","last_undegraded":"2024-03-26T21:34:39.975168+0000","last_fullsized":"2024-03-26T21:34:39.975168+0000","mapping_epoch":95,"log_start":"136'9600","ondisk_log_start":"136'9600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'14069","last_scrub_stamp":"2024-03-26T21:34:19.726617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:19.726617+0000","objects_scrubbed":720,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.017760+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14780,"num_write_kb":472960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078579+0000","last_change":"2024-03-26T21:32:30.548012+0000","last_active":"2024-03-26T21:34:40.078579+0000","last_peered":"2024-03-26T21:34:40.078579+0000","last_clean":"2024-03-26T21:34:40.078579+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:34:40.078579+0000","last_undegraded":"2024-03-26T21:34:40.078579+0000","last_fullsized":"2024-03-26T21:34:40.078579+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.547900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":260,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976088+0000","last_change":"2024-03-26T21:34:28.712452+0000","last_active":"2024-03-26T21:34:39.976088+0000","last_peered":"2024-03-26T21:34:39.976088+0000","last_clean":"2024-03-26T21:34:39.976088+0000","last_became_active":"2024-03-26T21:32:55.475860+0000","last_became_peered":"2024-03-26T21:32:55.475860+0000","last_unstale":"2024-03-26T21:34:39.976088+0000","last_undegraded":"2024-03-26T21:34:39.976088+0000","last_fullsized":"2024-03-26T21:34:39.976088+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:34:28.712368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.712368+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.654006+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":262,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077694+0000","last_change":"2024-03-26T21:34:32.761556+0000","last_active":"2024-03-26T21:34:40.077694+0000","last_peered":"2024-03-26T21:34:40.077694+0000","last_clean":"2024-03-26T21:34:40.077694+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:34:40.077694+0000","last_undegraded":"2024-03-26T21:34:40.077694+0000","last_fullsized":"2024-03-26T21:34:40.077694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:32.761425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.761425+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.953919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"134'10","reported_seq":318,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975503+0000","last_change":"2024-03-26T21:34:30.686544+0000","last_active":"2024-03-26T21:34:39.975503+0000","last_peered":"2024-03-26T21:34:39.975503+0000","last_clean":"2024-03-26T21:34:39.975503+0000","last_became_active":"2024-03-26T21:32:55.471127+0000","last_became_peered":"2024-03-26T21:32:55.471127+0000","last_unstale":"2024-03-26T21:34:39.975503+0000","last_undegraded":"2024-03-26T21:34:39.975503+0000","last_fullsized":"2024-03-26T21:34:39.975503+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:30.686411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.686411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.859040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"167'14906","reported_seq":15173,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078661+0000","last_change":"2024-03-26T21:34:06.998892+0000","last_active":"2024-03-26T21:34:40.078661+0000","last_peered":"2024-03-26T21:34:40.078661+0000","last_clean":"2024-03-26T21:34:40.078661+0000","last_became_active":"2024-03-26T21:30:38.548054+0000","last_became_peered":"2024-03-26T21:30:38.548054+0000","last_unstale":"2024-03-26T21:34:40.078661+0000","last_undegraded":"2024-03-26T21:34:40.078661+0000","last_fullsized":"2024-03-26T21:34:40.078661+0000","mapping_epoch":95,"log_start":"139'10200","ondisk_log_start":"139'10200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'13306","last_scrub_stamp":"2024-03-26T21:34:06.998566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:06.998566+0000","objects_scrubbed":1627,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:14.155892+0000","scrub_duration":783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14906,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078354+0000","last_change":"2024-03-26T21:32:42.225753+0000","last_active":"2024-03-26T21:34:40.078354+0000","last_peered":"2024-03-26T21:34:40.078354+0000","last_clean":"2024-03-26T21:34:40.078354+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:34:40.078354+0000","last_undegraded":"2024-03-26T21:34:40.078354+0000","last_fullsized":"2024-03-26T21:34:40.078354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:42.225645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:42.225645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284535+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":249,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975955+0000","last_change":"2024-03-26T21:33:16.562187+0000","last_active":"2024-03-26T21:34:39.975955+0000","last_peered":"2024-03-26T21:34:39.975955+0000","last_clean":"2024-03-26T21:34:39.975955+0000","last_became_active":"2024-03-26T21:32:55.471766+0000","last_became_peered":"2024-03-26T21:32:55.471766+0000","last_unstale":"2024-03-26T21:34:39.975955+0000","last_undegraded":"2024-03-26T21:34:39.975955+0000","last_fullsized":"2024-03-26T21:34:39.975955+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:33:16.562082+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:33:16.562082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.526159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":168,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920541+0000","last_change":"2024-03-26T21:34:18.762352+0000","last_active":"2024-03-26T21:34:33.920541+0000","last_peered":"2024-03-26T21:34:33.920541+0000","last_clean":"2024-03-26T21:34:33.920541+0000","last_became_active":"2024-03-26T21:32:55.467883+0000","last_became_peered":"2024-03-26T21:32:55.467883+0000","last_unstale":"2024-03-26T21:34:33.920541+0000","last_undegraded":"2024-03-26T21:34:33.920541+0000","last_fullsized":"2024-03-26T21:34:33.920541+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:18.762259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.762259+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.929171+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077316+0000","last_change":"2024-03-26T21:34:20.747940+0000","last_active":"2024-03-26T21:34:40.077316+0000","last_peered":"2024-03-26T21:34:40.077316+0000","last_clean":"2024-03-26T21:34:40.077316+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:34:40.077316+0000","last_undegraded":"2024-03-26T21:34:40.077316+0000","last_fullsized":"2024-03-26T21:34:40.077316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.747694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.747694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.007716+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"167'15266","reported_seq":21798,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974034+0000","last_change":"2024-03-26T21:34:39.974034+0000","last_active":"2024-03-26T21:34:39.974034+0000","last_peered":"2024-03-26T21:34:39.974034+0000","last_clean":"2024-03-26T21:34:39.974034+0000","last_became_active":"2024-03-26T21:33:50.930640+0000","last_became_peered":"2024-03-26T21:33:50.930640+0000","last_unstale":"2024-03-26T21:34:39.974034+0000","last_undegraded":"2024-03-26T21:34:39.974034+0000","last_fullsized":"2024-03-26T21:34:39.974034+0000","mapping_epoch":153,"log_start":"135'9718","ondisk_log_start":"135'9718","created":95,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":5548,"log_dups_size":0,"ondisk_log_size":5548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15266,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3178,"num_bytes_recovered":208273408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975142+0000","last_change":"2024-03-26T21:34:08.816123+0000","last_active":"2024-03-26T21:34:39.975142+0000","last_peered":"2024-03-26T21:34:39.975142+0000","last_clean":"2024-03-26T21:34:39.975142+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:34:39.975142+0000","last_undegraded":"2024-03-26T21:34:39.975142+0000","last_fullsized":"2024-03-26T21:34:39.975142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.816046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.816046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.663133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078600+0000","last_change":"2024-03-26T21:34:07.585143+0000","last_active":"2024-03-26T21:34:40.078600+0000","last_peered":"2024-03-26T21:34:40.078600+0000","last_clean":"2024-03-26T21:34:40.078600+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:34:40.078600+0000","last_undegraded":"2024-03-26T21:34:40.078600+0000","last_fullsized":"2024-03-26T21:34:40.078600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.585039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.585039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.578417+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":249,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975988+0000","last_change":"2024-03-26T21:34:26.757750+0000","last_active":"2024-03-26T21:34:39.975988+0000","last_peered":"2024-03-26T21:34:39.975988+0000","last_clean":"2024-03-26T21:34:39.975988+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:34:39.975988+0000","last_undegraded":"2024-03-26T21:34:39.975988+0000","last_fullsized":"2024-03-26T21:34:39.975988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.757669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.757669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:56.675099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":277,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078528+0000","last_change":"2024-03-26T21:33:25.213686+0000","last_active":"2024-03-26T21:34:40.078528+0000","last_peered":"2024-03-26T21:34:40.078528+0000","last_clean":"2024-03-26T21:34:40.078528+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:34:40.078528+0000","last_undegraded":"2024-03-26T21:34:40.078528+0000","last_fullsized":"2024-03-26T21:34:40.078528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:33:25.213564+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:47.289082+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"167'15092","reported_seq":24161,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077180+0000","last_change":"2024-03-26T21:34:18.073137+0000","last_active":"2024-03-26T21:34:40.077180+0000","last_peered":"2024-03-26T21:34:40.077180+0000","last_clean":"2024-03-26T21:34:40.077180+0000","last_became_active":"2024-03-26T21:33:30.904812+0000","last_became_peered":"2024-03-26T21:33:30.904812+0000","last_unstale":"2024-03-26T21:34:40.077180+0000","last_undegraded":"2024-03-26T21:34:40.077180+0000","last_fullsized":"2024-03-26T21:34:40.077180+0000","mapping_epoch":147,"log_start":"125'8100","ondisk_log_start":"125'8100","created":95,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'14216","last_scrub_stamp":"2024-03-26T21:34:18.072795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.072795+0000","objects_scrubbed":885,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.167086+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15092,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":291110912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974134+0000","last_change":"2024-03-26T21:31:37.722079+0000","last_active":"2024-03-26T21:34:39.974134+0000","last_peered":"2024-03-26T21:34:39.974134+0000","last_clean":"2024-03-26T21:34:39.974134+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:34:39.974134+0000","last_undegraded":"2024-03-26T21:34:39.974134+0000","last_fullsized":"2024-03-26T21:34:39.974134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.721821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016183+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":234,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078686+0000","last_change":"2024-03-26T21:33:29.574051+0000","last_active":"2024-03-26T21:34:40.078686+0000","last_peered":"2024-03-26T21:34:40.078686+0000","last_clean":"2024-03-26T21:34:40.078686+0000","last_became_active":"2024-03-26T21:32:55.471707+0000","last_became_peered":"2024-03-26T21:32:55.471707+0000","last_unstale":"2024-03-26T21:34:40.078686+0000","last_undegraded":"2024-03-26T21:34:40.078686+0000","last_fullsized":"2024-03-26T21:34:40.078686+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:29.573915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.573915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.880689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078457+0000","last_change":"2024-03-26T21:34:14.695183+0000","last_active":"2024-03-26T21:34:40.078457+0000","last_peered":"2024-03-26T21:34:40.078457+0000","last_clean":"2024-03-26T21:34:40.078457+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:34:40.078457+0000","last_undegraded":"2024-03-26T21:34:40.078457+0000","last_fullsized":"2024-03-26T21:34:40.078457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.695081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.695081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:17.363369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":246,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976263+0000","last_change":"2024-03-26T21:34:15.948871+0000","last_active":"2024-03-26T21:34:39.976263+0000","last_peered":"2024-03-26T21:34:39.976263+0000","last_clean":"2024-03-26T21:34:39.976263+0000","last_became_active":"2024-03-26T21:32:55.478663+0000","last_became_peered":"2024-03-26T21:32:55.478663+0000","last_unstale":"2024-03-26T21:34:39.976263+0000","last_undegraded":"2024-03-26T21:34:39.976263+0000","last_fullsized":"2024-03-26T21:34:39.976263+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:15.948730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.948730+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:17.502102+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"167'15154","reported_seq":15416,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079116+0000","last_change":"2024-03-26T21:34:10.537506+0000","last_active":"2024-03-26T21:34:40.079116+0000","last_peered":"2024-03-26T21:34:40.079116+0000","last_clean":"2024-03-26T21:34:40.079116+0000","last_became_active":"2024-03-26T21:30:38.548880+0000","last_became_peered":"2024-03-26T21:30:38.548880+0000","last_unstale":"2024-03-26T21:34:40.079116+0000","last_undegraded":"2024-03-26T21:34:40.079116+0000","last_fullsized":"2024-03-26T21:34:40.079116+0000","mapping_epoch":95,"log_start":"128'9012","ondisk_log_start":"128'9012","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'13814","last_scrub_stamp":"2024-03-26T21:34:10.537259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.537259+0000","objects_scrubbed":1352,"log_size":6142,"log_dups_size":0,"ondisk_log_size":6142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:22.787283+0000","scrub_duration":414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15154,"num_write_kb":484928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078482+0000","last_change":"2024-03-26T21:34:10.292343+0000","last_active":"2024-03-26T21:34:40.078482+0000","last_peered":"2024-03-26T21:34:40.078482+0000","last_clean":"2024-03-26T21:34:40.078482+0000","last_became_active":"2024-03-26T21:32:55.468927+0000","last_became_peered":"2024-03-26T21:32:55.468927+0000","last_unstale":"2024-03-26T21:34:40.078482+0000","last_undegraded":"2024-03-26T21:34:40.078482+0000","last_fullsized":"2024-03-26T21:34:40.078482+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:10.292229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.292229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:14.024010+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":250,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078278+0000","last_change":"2024-03-26T21:34:16.662556+0000","last_active":"2024-03-26T21:34:40.078278+0000","last_peered":"2024-03-26T21:34:40.078278+0000","last_clean":"2024-03-26T21:34:40.078278+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:34:40.078278+0000","last_undegraded":"2024-03-26T21:34:40.078278+0000","last_fullsized":"2024-03-26T21:34:40.078278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:34:16.662471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:16.662471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.912431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":147,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920397+0000","last_change":"2024-03-26T21:32:55.480927+0000","last_active":"2024-03-26T21:34:33.920397+0000","last_peered":"2024-03-26T21:34:33.920397+0000","last_clean":"2024-03-26T21:34:33.920397+0000","last_became_active":"2024-03-26T21:32:55.480439+0000","last_became_peered":"2024-03-26T21:32:55.480439+0000","last_unstale":"2024-03-26T21:34:33.920397+0000","last_undegraded":"2024-03-26T21:34:33.920397+0000","last_fullsized":"2024-03-26T21:34:33.920397+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:15.290476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:15.290476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":313,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077924+0000","last_change":"2024-03-26T21:34:13.674583+0000","last_active":"2024-03-26T21:34:40.077924+0000","last_peered":"2024-03-26T21:34:40.077924+0000","last_clean":"2024-03-26T21:34:40.077924+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:34:40.077924+0000","last_undegraded":"2024-03-26T21:34:40.077924+0000","last_fullsized":"2024-03-26T21:34:40.077924+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:34:13.674406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.674406+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.518466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":234,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974594+0000","last_change":"2024-03-26T21:34:18.955506+0000","last_active":"2024-03-26T21:34:39.974594+0000","last_peered":"2024-03-26T21:34:39.974594+0000","last_clean":"2024-03-26T21:34:39.974594+0000","last_became_active":"2024-03-26T21:32:55.478576+0000","last_became_peered":"2024-03-26T21:32:55.478576+0000","last_unstale":"2024-03-26T21:34:39.974594+0000","last_undegraded":"2024-03-26T21:34:39.974594+0000","last_fullsized":"2024-03-26T21:34:39.974594+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:18.955400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.955400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.675127+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"167'14980","reported_seq":24085,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974487+0000","last_change":"2024-03-26T21:34:32.090977+0000","last_active":"2024-03-26T21:34:39.974487+0000","last_peered":"2024-03-26T21:34:39.974487+0000","last_clean":"2024-03-26T21:34:39.974487+0000","last_became_active":"2024-03-26T21:33:30.912383+0000","last_became_peered":"2024-03-26T21:33:30.912383+0000","last_unstale":"2024-03-26T21:34:39.974487+0000","last_undegraded":"2024-03-26T21:34:39.974487+0000","last_fullsized":"2024-03-26T21:34:39.974487+0000","mapping_epoch":147,"log_start":"135'9468","ondisk_log_start":"135'9468","created":95,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'14980","last_scrub_stamp":"2024-03-26T21:34:32.090869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.090869+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:49.667580+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14980,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4465,"num_bytes_recovered":292618240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976129+0000","last_change":"2024-03-26T21:33:28.620552+0000","last_active":"2024-03-26T21:34:39.976129+0000","last_peered":"2024-03-26T21:34:39.976129+0000","last_clean":"2024-03-26T21:34:39.976129+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:34:39.976129+0000","last_undegraded":"2024-03-26T21:34:39.976129+0000","last_fullsized":"2024-03-26T21:34:39.976129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:28.620444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:33:28.620444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":250,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975391+0000","last_change":"2024-03-26T21:34:24.599907+0000","last_active":"2024-03-26T21:34:39.975391+0000","last_peered":"2024-03-26T21:34:39.975391+0000","last_clean":"2024-03-26T21:34:39.975391+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:34:39.975391+0000","last_undegraded":"2024-03-26T21:34:39.975391+0000","last_fullsized":"2024-03-26T21:34:39.975391+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.599655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.967423+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":248,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077434+0000","last_change":"2024-03-26T21:34:29.929261+0000","last_active":"2024-03-26T21:34:40.077434+0000","last_peered":"2024-03-26T21:34:40.077434+0000","last_clean":"2024-03-26T21:34:40.077434+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:34:40.077434+0000","last_undegraded":"2024-03-26T21:34:40.077434+0000","last_fullsized":"2024-03-26T21:34:40.077434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:34:29.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.201699+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":269,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975908+0000","last_change":"2024-03-26T21:34:10.523681+0000","last_active":"2024-03-26T21:34:39.975908+0000","last_peered":"2024-03-26T21:34:39.975908+0000","last_clean":"2024-03-26T21:34:39.975908+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:34:39.975908+0000","last_undegraded":"2024-03-26T21:34:39.975908+0000","last_fullsized":"2024-03-26T21:34:39.975908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:34:10.523547+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.523547+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.692847+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078751+0000","last_change":"2024-03-26T21:34:32.868281+0000","last_active":"2024-03-26T21:34:40.078751+0000","last_peered":"2024-03-26T21:34:40.078751+0000","last_clean":"2024-03-26T21:34:40.078751+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:34:40.078751+0000","last_undegraded":"2024-03-26T21:34:40.078751+0000","last_fullsized":"2024-03-26T21:34:40.078751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.868200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.530052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"167'14834","reported_seq":15075,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975917+0000","last_change":"2024-03-26T21:32:22.439431+0000","last_active":"2024-03-26T21:34:39.975917+0000","last_peered":"2024-03-26T21:34:39.975917+0000","last_clean":"2024-03-26T21:34:39.975917+0000","last_became_active":"2024-03-26T21:30:38.550658+0000","last_became_peered":"2024-03-26T21:30:38.550658+0000","last_unstale":"2024-03-26T21:34:39.975917+0000","last_undegraded":"2024-03-26T21:34:39.975917+0000","last_fullsized":"2024-03-26T21:34:39.975917+0000","mapping_epoch":95,"log_start":"131'9316","ondisk_log_start":"131'9316","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7328","last_scrub_stamp":"2024-03-26T21:32:22.439030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.439030+0000","objects_scrubbed":7259,"log_size":5518,"log_dups_size":0,"ondisk_log_size":5518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016333+0000","scrub_duration":3392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14834,"num_write_kb":474688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078228+0000","last_change":"2024-03-26T21:33:26.736579+0000","last_active":"2024-03-26T21:34:40.078228+0000","last_peered":"2024-03-26T21:34:40.078228+0000","last_clean":"2024-03-26T21:34:40.078228+0000","last_became_active":"2024-03-26T21:32:55.474113+0000","last_became_peered":"2024-03-26T21:32:55.474113+0000","last_unstale":"2024-03-26T21:34:40.078228+0000","last_undegraded":"2024-03-26T21:34:40.078228+0000","last_fullsized":"2024-03-26T21:34:40.078228+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:26.736456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:26.736456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":494,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975233+0000","last_change":"2024-03-26T21:34:17.683647+0000","last_active":"2024-03-26T21:34:39.975233+0000","last_peered":"2024-03-26T21:34:39.975233+0000","last_clean":"2024-03-26T21:34:39.975233+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:34:39.975233+0000","last_undegraded":"2024-03-26T21:34:39.975233+0000","last_fullsized":"2024-03-26T21:34:39.975233+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:34:17.683557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:34:17.683557+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.639537+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078559+0000","last_change":"2024-03-26T21:34:28.378904+0000","last_active":"2024-03-26T21:34:40.078559+0000","last_peered":"2024-03-26T21:34:40.078559+0000","last_clean":"2024-03-26T21:34:40.078559+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:34:40.078559+0000","last_undegraded":"2024-03-26T21:34:40.078559+0000","last_fullsized":"2024-03-26T21:34:40.078559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:28.378787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.378787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:34.024112+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"134'26","reported_seq":352,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.079085+0000","last_change":"2024-03-26T21:34:31.976293+0000","last_active":"2024-03-26T21:34:40.079085+0000","last_peered":"2024-03-26T21:34:40.079085+0000","last_clean":"2024-03-26T21:34:40.079085+0000","last_became_active":"2024-03-26T21:32:55.477984+0000","last_became_peered":"2024-03-26T21:32:55.477984+0000","last_unstale":"2024-03-26T21:34:40.079085+0000","last_undegraded":"2024-03-26T21:34:40.079085+0000","last_fullsized":"2024-03-26T21:34:40.079085+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'26","last_scrub_stamp":"2024-03-26T21:34:31.976190+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.976190+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.446068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975775+0000","last_change":"2024-03-26T21:34:20.660945+0000","last_active":"2024-03-26T21:34:39.975775+0000","last_peered":"2024-03-26T21:34:39.975775+0000","last_clean":"2024-03-26T21:34:39.975775+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:34:39.975775+0000","last_undegraded":"2024-03-26T21:34:39.975775+0000","last_fullsized":"2024-03-26T21:34:39.975775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.660825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.660825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.066534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"167'14820","reported_seq":15079,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975732+0000","last_change":"2024-03-26T21:34:13.208418+0000","last_active":"2024-03-26T21:34:39.975732+0000","last_peered":"2024-03-26T21:34:39.975732+0000","last_clean":"2024-03-26T21:34:39.975732+0000","last_became_active":"2024-03-26T21:30:38.543531+0000","last_became_peered":"2024-03-26T21:30:38.543531+0000","last_unstale":"2024-03-26T21:34:39.975732+0000","last_undegraded":"2024-03-26T21:34:39.975732+0000","last_fullsized":"2024-03-26T21:34:39.975732+0000","mapping_epoch":95,"log_start":"135'9500","ondisk_log_start":"135'9500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'13707","last_scrub_stamp":"2024-03-26T21:34:13.208218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.208218+0000","objects_scrubbed":1124,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:34.907350+0000","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14820,"num_write_kb":474240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975602+0000","last_change":"2024-03-26T21:34:14.558500+0000","last_active":"2024-03-26T21:34:39.975602+0000","last_peered":"2024-03-26T21:34:39.975602+0000","last_clean":"2024-03-26T21:34:39.975602+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:34:39.975602+0000","last_undegraded":"2024-03-26T21:34:39.975602+0000","last_fullsized":"2024-03-26T21:34:39.975602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.557934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.557934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.351356+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":242,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974379+0000","last_change":"2024-03-26T21:34:25.034122+0000","last_active":"2024-03-26T21:34:39.974379+0000","last_peered":"2024-03-26T21:34:39.974379+0000","last_clean":"2024-03-26T21:34:39.974379+0000","last_became_active":"2024-03-26T21:32:55.475773+0000","last_became_peered":"2024-03-26T21:32:55.475773+0000","last_unstale":"2024-03-26T21:34:39.974379+0000","last_undegraded":"2024-03-26T21:34:39.974379+0000","last_fullsized":"2024-03-26T21:34:39.974379+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.033987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.033987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:25.283140+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":612,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975249+0000","last_change":"2024-03-26T21:33:29.586212+0000","last_active":"2024-03-26T21:34:39.975249+0000","last_peered":"2024-03-26T21:34:39.975249+0000","last_clean":"2024-03-26T21:34:39.975249+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:34:39.975249+0000","last_undegraded":"2024-03-26T21:34:39.975249+0000","last_fullsized":"2024-03-26T21:34:39.975249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:33:29.586133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.586133+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.990517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":360,"num_read_kb":355,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":275,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974813+0000","last_change":"2024-03-26T21:32:39.859396+0000","last_active":"2024-03-26T21:34:39.974813+0000","last_peered":"2024-03-26T21:34:39.974813+0000","last_clean":"2024-03-26T21:34:39.974813+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:34:39.974813+0000","last_undegraded":"2024-03-26T21:34:39.974813+0000","last_fullsized":"2024-03-26T21:34:39.974813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:32:39.858918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:39.858918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016371+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078704+0000","last_change":"2024-03-26T21:34:09.662284+0000","last_active":"2024-03-26T21:34:40.078704+0000","last_peered":"2024-03-26T21:34:40.078704+0000","last_clean":"2024-03-26T21:34:40.078704+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:34:40.078704+0000","last_undegraded":"2024-03-26T21:34:40.078704+0000","last_fullsized":"2024-03-26T21:34:40.078704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.662190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.662190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.442388+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"167'14972","reported_seq":15208,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077562+0000","last_change":"2024-03-26T21:32:36.082531+0000","last_active":"2024-03-26T21:34:40.077562+0000","last_peered":"2024-03-26T21:34:40.077562+0000","last_clean":"2024-03-26T21:34:40.077562+0000","last_became_active":"2024-03-26T21:30:38.543747+0000","last_became_peered":"2024-03-26T21:30:38.543747+0000","last_unstale":"2024-03-26T21:34:40.077562+0000","last_undegraded":"2024-03-26T21:34:40.077562+0000","last_fullsized":"2024-03-26T21:34:40.077562+0000","mapping_epoch":95,"log_start":"131'9341","ondisk_log_start":"131'9341","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'8255","last_scrub_stamp":"2024-03-26T21:32:36.082442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.082442+0000","objects_scrubbed":6758,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790309+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14972,"num_write_kb":479104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975193+0000","last_change":"2024-03-26T21:34:08.527352+0000","last_active":"2024-03-26T21:34:39.975193+0000","last_peered":"2024-03-26T21:34:39.975193+0000","last_clean":"2024-03-26T21:34:39.975193+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:34:39.975193+0000","last_undegraded":"2024-03-26T21:34:39.975193+0000","last_fullsized":"2024-03-26T21:34:39.975193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.527230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.527230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.546653+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975844+0000","last_change":"2024-03-26T21:34:09.554532+0000","last_active":"2024-03-26T21:34:39.975844+0000","last_peered":"2024-03-26T21:34:39.975844+0000","last_clean":"2024-03-26T21:34:39.975844+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:34:39.975844+0000","last_undegraded":"2024-03-26T21:34:39.975844+0000","last_fullsized":"2024-03-26T21:34:39.975844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.554099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.554099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:12.804063+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":265,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974850+0000","last_change":"2024-03-26T21:31:32.754170+0000","last_active":"2024-03-26T21:34:39.974850+0000","last_peered":"2024-03-26T21:34:39.974850+0000","last_clean":"2024-03-26T21:34:39.974850+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:34:39.974850+0000","last_undegraded":"2024-03-26T21:34:39.974850+0000","last_fullsized":"2024-03-26T21:34:39.974850+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:31:32.754010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.754010+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016130+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078535+0000","last_change":"2024-03-26T21:33:24.229284+0000","last_active":"2024-03-26T21:34:40.078535+0000","last_peered":"2024-03-26T21:34:40.078535+0000","last_clean":"2024-03-26T21:34:40.078535+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:34:40.078535+0000","last_undegraded":"2024-03-26T21:34:40.078535+0000","last_fullsized":"2024-03-26T21:34:40.078535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:33:24.229192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:45.347268+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"167'14744","reported_seq":21354,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.919971+0000","last_change":"2024-03-26T21:33:50.324114+0000","last_active":"2024-03-26T21:34:33.919971+0000","last_peered":"2024-03-26T21:34:33.919971+0000","last_clean":"2024-03-26T21:34:33.919971+0000","last_became_active":"2024-03-26T21:33:50.264341+0000","last_became_peered":"2024-03-26T21:33:50.264341+0000","last_unstale":"2024-03-26T21:34:33.919971+0000","last_undegraded":"2024-03-26T21:34:33.919971+0000","last_fullsized":"2024-03-26T21:34:33.919971+0000","mapping_epoch":152,"log_start":"138'9900","ondisk_log_start":"138'9900","created":95,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:37.495090+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14744,"num_write_kb":471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3221,"num_bytes_recovered":210960384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":221,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.973781+0000","last_change":"2024-03-26T21:32:13.010288+0000","last_active":"2024-03-26T21:34:39.973781+0000","last_peered":"2024-03-26T21:34:39.973781+0000","last_clean":"2024-03-26T21:34:39.973781+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:34:39.973781+0000","last_undegraded":"2024-03-26T21:34:39.973781+0000","last_fullsized":"2024-03-26T21:34:39.973781+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:13.010170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.010170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016248+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078995+0000","last_change":"2024-03-26T21:32:35.525387+0000","last_active":"2024-03-26T21:34:40.078995+0000","last_peered":"2024-03-26T21:34:40.078995+0000","last_clean":"2024-03-26T21:34:40.078995+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:34:40.078995+0000","last_undegraded":"2024-03-26T21:34:40.078995+0000","last_fullsized":"2024-03-26T21:34:40.078995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:32:35.525298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:43.790343+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":253,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975848+0000","last_change":"2024-03-26T21:33:19.746237+0000","last_active":"2024-03-26T21:34:39.975848+0000","last_peered":"2024-03-26T21:34:39.975848+0000","last_clean":"2024-03-26T21:34:39.975848+0000","last_became_active":"2024-03-26T21:32:55.465412+0000","last_became_peered":"2024-03-26T21:32:55.465412+0000","last_unstale":"2024-03-26T21:34:39.975848+0000","last_undegraded":"2024-03-26T21:34:39.975848+0000","last_fullsized":"2024-03-26T21:34:39.975848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:33:19.745702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:33:19.745702+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:41.674016+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":307,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974584+0000","last_change":"2024-03-26T21:32:11.923454+0000","last_active":"2024-03-26T21:34:39.974584+0000","last_peered":"2024-03-26T21:34:39.974584+0000","last_clean":"2024-03-26T21:34:39.974584+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:34:39.974584+0000","last_undegraded":"2024-03-26T21:34:39.974584+0000","last_fullsized":"2024-03-26T21:34:39.974584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:11.916784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:11.916784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"167'14820","reported_seq":15055,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078507+0000","last_change":"2024-03-26T21:32:37.145445+0000","last_active":"2024-03-26T21:34:40.078507+0000","last_peered":"2024-03-26T21:34:40.078507+0000","last_clean":"2024-03-26T21:34:40.078507+0000","last_became_active":"2024-03-26T21:30:38.549806+0000","last_became_peered":"2024-03-26T21:30:38.549806+0000","last_unstale":"2024-03-26T21:34:40.078507+0000","last_undegraded":"2024-03-26T21:34:40.078507+0000","last_fullsized":"2024-03-26T21:34:40.078507+0000","mapping_epoch":95,"log_start":"139'10200","ondisk_log_start":"139'10200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'8279","last_scrub_stamp":"2024-03-26T21:32:37.145169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:37.145169+0000","objects_scrubbed":6608,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14820,"num_write_kb":474240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.975576+0000","last_change":"2024-03-26T21:33:30.581306+0000","last_active":"2024-03-26T21:34:39.975576+0000","last_peered":"2024-03-26T21:34:39.975576+0000","last_clean":"2024-03-26T21:34:39.975576+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:34:39.975576+0000","last_undegraded":"2024-03-26T21:34:39.975576+0000","last_fullsized":"2024-03-26T21:34:39.975576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.581234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.581234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.788016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":235,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974918+0000","last_change":"2024-03-26T21:34:10.057922+0000","last_active":"2024-03-26T21:34:39.974918+0000","last_peered":"2024-03-26T21:34:39.974918+0000","last_clean":"2024-03-26T21:34:39.974918+0000","last_became_active":"2024-03-26T21:32:55.479274+0000","last_became_peered":"2024-03-26T21:32:55.479274+0000","last_unstale":"2024-03-26T21:34:39.974918+0000","last_undegraded":"2024-03-26T21:34:39.974918+0000","last_fullsized":"2024-03-26T21:34:39.974918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:34:10.057797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.057797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:23.442880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":164,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920429+0000","last_change":"2024-03-26T21:32:55.472003+0000","last_active":"2024-03-26T21:34:33.920429+0000","last_peered":"2024-03-26T21:34:33.920429+0000","last_clean":"2024-03-26T21:34:33.920429+0000","last_became_active":"2024-03-26T21:32:55.469472+0000","last_became_peered":"2024-03-26T21:32:55.469472+0000","last_unstale":"2024-03-26T21:34:33.920429+0000","last_undegraded":"2024-03-26T21:34:33.920429+0000","last_fullsized":"2024-03-26T21:34:33.920429+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"134'10","reported_seq":319,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078248+0000","last_change":"2024-03-26T21:34:12.341709+0000","last_active":"2024-03-26T21:34:40.078248+0000","last_peered":"2024-03-26T21:34:40.078248+0000","last_clean":"2024-03-26T21:34:40.078248+0000","last_became_active":"2024-03-26T21:32:55.476581+0000","last_became_peered":"2024-03-26T21:32:55.476581+0000","last_unstale":"2024-03-26T21:34:40.078248+0000","last_undegraded":"2024-03-26T21:34:40.078248+0000","last_fullsized":"2024-03-26T21:34:40.078248+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:12.341310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.341310+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.217651+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"167'14938","reported_seq":24046,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078448+0000","last_change":"2024-03-26T21:34:17.741908+0000","last_active":"2024-03-26T21:34:40.078448+0000","last_peered":"2024-03-26T21:34:40.078448+0000","last_clean":"2024-03-26T21:34:40.078448+0000","last_became_active":"2024-03-26T21:33:29.900183+0000","last_became_peered":"2024-03-26T21:33:29.900183+0000","last_unstale":"2024-03-26T21:34:40.078448+0000","last_undegraded":"2024-03-26T21:34:40.078448+0000","last_fullsized":"2024-03-26T21:34:40.078448+0000","mapping_epoch":146,"log_start":"139'10300","ondisk_log_start":"139'10300","created":95,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'14041","last_scrub_stamp":"2024-03-26T21:34:17.741512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:17.741512+0000","objects_scrubbed":899,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.228133+0000","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14938,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4461,"num_bytes_recovered":292356096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":137,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920290+0000","last_change":"2024-03-26T21:34:33.905393+0000","last_active":"2024-03-26T21:34:33.920290+0000","last_peered":"2024-03-26T21:34:33.920290+0000","last_clean":"2024-03-26T21:34:33.920290+0000","last_became_active":"2024-03-26T21:32:55.476227+0000","last_became_peered":"2024-03-26T21:32:55.476227+0000","last_unstale":"2024-03-26T21:34:33.920290+0000","last_undegraded":"2024-03-26T21:34:33.920290+0000","last_fullsized":"2024-03-26T21:34:33.920290+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.905276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.905276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.967842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078807+0000","last_change":"2024-03-26T21:32:17.198343+0000","last_active":"2024-03-26T21:34:40.078807+0000","last_peered":"2024-03-26T21:34:40.078807+0000","last_clean":"2024-03-26T21:34:40.078807+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:34:40.078807+0000","last_undegraded":"2024-03-26T21:34:40.078807+0000","last_fullsized":"2024-03-26T21:34:40.078807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:17.198228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:17.198228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284434+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":262,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078109+0000","last_change":"2024-03-26T21:32:33.337863+0000","last_active":"2024-03-26T21:34:40.078109+0000","last_peered":"2024-03-26T21:34:40.078109+0000","last_clean":"2024-03-26T21:34:40.078109+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:34:40.078109+0000","last_undegraded":"2024-03-26T21:34:40.078109+0000","last_fullsized":"2024-03-26T21:34:40.078109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:32:33.337581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:33.337581+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":98,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920063+0000","last_change":"2024-03-26T21:34:07.791992+0000","last_active":"2024-03-26T21:34:33.920063+0000","last_peered":"2024-03-26T21:34:33.920063+0000","last_clean":"2024-03-26T21:34:33.920063+0000","last_became_active":"2024-03-26T21:32:55.468331+0000","last_became_peered":"2024-03-26T21:32:55.468331+0000","last_unstale":"2024-03-26T21:34:33.920063+0000","last_undegraded":"2024-03-26T21:34:33.920063+0000","last_fullsized":"2024-03-26T21:34:33.920063+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.791763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.791763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.911000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"167'15014","reported_seq":15272,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078554+0000","last_change":"2024-03-26T21:34:16.454516+0000","last_active":"2024-03-26T21:34:40.078554+0000","last_peered":"2024-03-26T21:34:40.078554+0000","last_clean":"2024-03-26T21:34:40.078554+0000","last_became_active":"2024-03-26T21:30:38.539838+0000","last_became_peered":"2024-03-26T21:30:38.539838+0000","last_unstale":"2024-03-26T21:34:40.078554+0000","last_undegraded":"2024-03-26T21:34:40.078554+0000","last_fullsized":"2024-03-26T21:34:40.078554+0000","mapping_epoch":95,"log_start":"128'8860","ondisk_log_start":"128'8860","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'14015","last_scrub_stamp":"2024-03-26T21:34:16.454384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:16.454384+0000","objects_scrubbed":1013,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.796897+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15014,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078076+0000","last_change":"2024-03-26T21:34:15.695258+0000","last_active":"2024-03-26T21:34:40.078076+0000","last_peered":"2024-03-26T21:34:40.078076+0000","last_clean":"2024-03-26T21:34:40.078076+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:34:40.078076+0000","last_undegraded":"2024-03-26T21:34:40.078076+0000","last_fullsized":"2024-03-26T21:34:40.078076+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:15.695126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.695126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.175611+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"134'10","reported_seq":188,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.921333+0000","last_change":"2024-03-26T21:32:55.480444+0000","last_active":"2024-03-26T21:34:33.921333+0000","last_peered":"2024-03-26T21:34:33.921333+0000","last_clean":"2024-03-26T21:34:33.921333+0000","last_became_active":"2024-03-26T21:32:55.478624+0000","last_became_peered":"2024-03-26T21:32:55.478624+0000","last_unstale":"2024-03-26T21:34:33.921333+0000","last_undegraded":"2024-03-26T21:34:33.921333+0000","last_fullsized":"2024-03-26T21:34:33.921333+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.407216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.407216+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:39.910167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":233,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077966+0000","last_change":"2024-03-26T21:31:29.955815+0000","last_active":"2024-03-26T21:34:40.077966+0000","last_peered":"2024-03-26T21:34:40.077966+0000","last_clean":"2024-03-26T21:34:40.077966+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:34:40.077966+0000","last_undegraded":"2024-03-26T21:34:40.077966+0000","last_fullsized":"2024-03-26T21:34:40.077966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.284317+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":271,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976070+0000","last_change":"2024-03-26T21:34:33.587083+0000","last_active":"2024-03-26T21:34:39.976070+0000","last_peered":"2024-03-26T21:34:39.976070+0000","last_clean":"2024-03-26T21:34:39.976070+0000","last_became_active":"2024-03-26T21:32:55.471751+0000","last_became_peered":"2024-03-26T21:32:55.471751+0000","last_unstale":"2024-03-26T21:34:39.976070+0000","last_undegraded":"2024-03-26T21:34:39.976070+0000","last_fullsized":"2024-03-26T21:34:39.976070+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:33.586983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.586983+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.630361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"167'14794","reported_seq":15050,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078701+0000","last_change":"2024-03-26T21:34:20.312619+0000","last_active":"2024-03-26T21:34:40.078701+0000","last_peered":"2024-03-26T21:34:40.078701+0000","last_clean":"2024-03-26T21:34:40.078701+0000","last_became_active":"2024-03-26T21:30:38.543981+0000","last_became_peered":"2024-03-26T21:30:38.543981+0000","last_unstale":"2024-03-26T21:34:40.078701+0000","last_undegraded":"2024-03-26T21:34:40.078701+0000","last_fullsized":"2024-03-26T21:34:40.078701+0000","mapping_epoch":95,"log_start":"127'8637","ondisk_log_start":"127'8637","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'14124","last_scrub_stamp":"2024-03-26T21:34:20.312393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.312393+0000","objects_scrubbed":681,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:42.106319+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14794,"num_write_kb":473345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077769+0000","last_change":"2024-03-26T21:34:32.360553+0000","last_active":"2024-03-26T21:34:40.077769+0000","last_peered":"2024-03-26T21:34:40.077769+0000","last_clean":"2024-03-26T21:34:40.077769+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:34:40.077769+0000","last_undegraded":"2024-03-26T21:34:40.077769+0000","last_fullsized":"2024-03-26T21:34:40.077769+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.360471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.360471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.149069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":226,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078606+0000","last_change":"2024-03-26T21:33:17.842797+0000","last_active":"2024-03-26T21:34:40.078606+0000","last_peered":"2024-03-26T21:34:40.078606+0000","last_clean":"2024-03-26T21:34:40.078606+0000","last_became_active":"2024-03-26T21:32:55.465348+0000","last_became_peered":"2024-03-26T21:32:55.465348+0000","last_unstale":"2024-03-26T21:34:40.078606+0000","last_undegraded":"2024-03-26T21:34:40.078606+0000","last_fullsized":"2024-03-26T21:34:40.078606+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:17.842709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:17.842709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.593643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":149,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920216+0000","last_change":"2024-03-26T21:34:24.836203+0000","last_active":"2024-03-26T21:34:33.920216+0000","last_peered":"2024-03-26T21:34:33.920216+0000","last_clean":"2024-03-26T21:34:33.920216+0000","last_became_active":"2024-03-26T21:32:55.463332+0000","last_became_peered":"2024-03-26T21:32:55.463332+0000","last_unstale":"2024-03-26T21:34:33.920216+0000","last_undegraded":"2024-03-26T21:34:33.920216+0000","last_fullsized":"2024-03-26T21:34:33.920216+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.835878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.835878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.538447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":160,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920648+0000","last_change":"2024-03-26T21:32:55.472726+0000","last_active":"2024-03-26T21:34:33.920648+0000","last_peered":"2024-03-26T21:34:33.920648+0000","last_clean":"2024-03-26T21:34:33.920648+0000","last_became_active":"2024-03-26T21:32:55.471887+0000","last_became_peered":"2024-03-26T21:32:55.471887+0000","last_unstale":"2024-03-26T21:34:33.920648+0000","last_undegraded":"2024-03-26T21:34:33.920648+0000","last_fullsized":"2024-03-26T21:34:33.920648+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"167'14998","reported_seq":15237,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078957+0000","last_change":"2024-03-26T21:32:16.545631+0000","last_active":"2024-03-26T21:34:40.078957+0000","last_peered":"2024-03-26T21:34:40.078957+0000","last_clean":"2024-03-26T21:34:40.078957+0000","last_became_active":"2024-03-26T21:30:38.545485+0000","last_became_peered":"2024-03-26T21:30:38.545485+0000","last_unstale":"2024-03-26T21:34:40.078957+0000","last_undegraded":"2024-03-26T21:34:40.078957+0000","last_fullsized":"2024-03-26T21:34:40.078957+0000","mapping_epoch":95,"log_start":"125'8000","ondisk_log_start":"125'8000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'7144","last_scrub_stamp":"2024-03-26T21:32:16.545400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:16.545400+0000","objects_scrubbed":7077,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.880652+0000","scrub_duration":2977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14998,"num_write_kb":479936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":230,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077701+0000","last_change":"2024-03-26T21:34:12.660667+0000","last_active":"2024-03-26T21:34:40.077701+0000","last_peered":"2024-03-26T21:34:40.077701+0000","last_clean":"2024-03-26T21:34:40.077701+0000","last_became_active":"2024-03-26T21:32:55.470413+0000","last_became_peered":"2024-03-26T21:32:55.470413+0000","last_unstale":"2024-03-26T21:34:40.077701+0000","last_undegraded":"2024-03-26T21:34:40.077701+0000","last_fullsized":"2024-03-26T21:34:40.077701+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:12.660555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.660555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.356026+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":243,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.976000+0000","last_change":"2024-03-26T21:34:26.592280+0000","last_active":"2024-03-26T21:34:39.976000+0000","last_peered":"2024-03-26T21:34:39.976000+0000","last_clean":"2024-03-26T21:34:39.976000+0000","last_became_active":"2024-03-26T21:32:55.472013+0000","last_became_peered":"2024-03-26T21:32:55.472013+0000","last_unstale":"2024-03-26T21:34:39.976000+0000","last_undegraded":"2024-03-26T21:34:39.976000+0000","last_fullsized":"2024-03-26T21:34:39.976000+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.591961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.591961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.616850+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":250,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.078227+0000","last_change":"2024-03-26T21:34:33.006206+0000","last_active":"2024-03-26T21:34:40.078227+0000","last_peered":"2024-03-26T21:34:40.078227+0000","last_clean":"2024-03-26T21:34:40.078227+0000","last_became_active":"2024-03-26T21:32:55.469205+0000","last_became_peered":"2024-03-26T21:32:55.469205+0000","last_unstale":"2024-03-26T21:34:40.078227+0000","last_undegraded":"2024-03-26T21:34:40.078227+0000","last_fullsized":"2024-03-26T21:34:40.078227+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.006101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.004115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":172,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920810+0000","last_change":"2024-03-26T21:32:55.464891+0000","last_active":"2024-03-26T21:34:33.920810+0000","last_peered":"2024-03-26T21:34:33.920810+0000","last_clean":"2024-03-26T21:34:33.920810+0000","last_became_active":"2024-03-26T21:32:55.464394+0000","last_became_peered":"2024-03-26T21:32:55.464394+0000","last_unstale":"2024-03-26T21:34:33.920810+0000","last_undegraded":"2024-03-26T21:34:33.920810+0000","last_fullsized":"2024-03-26T21:34:33.920810+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:30:19.267430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:19.267430+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"167'14858","reported_seq":15099,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:39.974819+0000","last_change":"2024-03-26T21:32:23.622753+0000","last_active":"2024-03-26T21:34:39.974819+0000","last_peered":"2024-03-26T21:34:39.974819+0000","last_clean":"2024-03-26T21:34:39.974819+0000","last_became_active":"2024-03-26T21:30:38.550667+0000","last_became_peered":"2024-03-26T21:30:38.550667+0000","last_unstale":"2024-03-26T21:34:39.974819+0000","last_undegraded":"2024-03-26T21:34:39.974819+0000","last_fullsized":"2024-03-26T21:34:39.974819+0000","mapping_epoch":95,"log_start":"131'9329","ondisk_log_start":"131'9329","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'7423","last_scrub_stamp":"2024-03-26T21:32:23.622299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:37.495090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.622299+0000","objects_scrubbed":7320,"log_size":5529,"log_dups_size":0,"ondisk_log_size":5529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:40.016315+0000","scrub_duration":3638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14858,"num_write_kb":475456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":133,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.920865+0000","last_change":"2024-03-26T21:32:55.472265+0000","last_active":"2024-03-26T21:34:33.920865+0000","last_peered":"2024-03-26T21:34:33.920865+0000","last_clean":"2024-03-26T21:34:33.920865+0000","last_became_active":"2024-03-26T21:32:55.471347+0000","last_became_peered":"2024-03-26T21:32:55.471347+0000","last_unstale":"2024-03-26T21:34:33.920865+0000","last_undegraded":"2024-03-26T21:34:33.920865+0000","last_fullsized":"2024-03-26T21:34:33.920865+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":315,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077406+0000","last_change":"2024-03-26T21:34:19.761741+0000","last_active":"2024-03-26T21:34:40.077406+0000","last_peered":"2024-03-26T21:34:40.077406+0000","last_clean":"2024-03-26T21:34:40.077406+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:34:40.077406+0000","last_undegraded":"2024-03-26T21:34:40.077406+0000","last_fullsized":"2024-03-26T21:34:40.077406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:34:19.761382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:19.761382+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.563424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":234,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-26T21:34:40.077889+0000","last_change":"2024-03-26T21:34:07.273064+0000","last_active":"2024-03-26T21:34:40.077889+0000","last_peered":"2024-03-26T21:34:40.077889+0000","last_clean":"2024-03-26T21:34:40.077889+0000","last_became_active":"2024-03-26T21:32:55.473354+0000","last_became_peered":"2024-03-26T21:32:55.473354+0000","last_unstale":"2024-03-26T21:34:40.077889+0000","last_undegraded":"2024-03-26T21:34:40.077889+0000","last_fullsized":"2024-03-26T21:34:40.077889+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.272948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.272948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:31.339263+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":163,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-26T21:34:33.921309+0000","last_change":"2024-03-26T21:34:14.786736+0000","last_active":"2024-03-26T21:34:33.921309+0000","last_peered":"2024-03-26T21:34:33.921309+0000","last_clean":"2024-03-26T21:34:33.921309+0000","last_became_active":"2024-03-26T21:32:55.476367+0000","last_became_peered":"2024-03-26T21:32:55.476367+0000","last_unstale":"2024-03-26T21:34:33.921309+0000","last_undegraded":"2024-03-26T21:34:33.921309+0000","last_fullsized":"2024-03-26T21:34:33.921309+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:34:14.786394+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.786394+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.339829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238970,"num_write_kb":7646977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19767,"num_bytes_recovered":1295319040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89489,"ondisk_log_size":89489,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":360,"num_read_kb":355,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070542,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123100,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622052,"statfs":{"total":95995035648,"available":95868981248,"internally_reserved":0,"allocated":1536000,"data_stored":1080988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364884,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123068,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622084,"statfs":{"total":95995035648,"available":95869014016,"internally_reserved":0,"allocated":1503232,"data_stored":1078576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148908,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596244,"statfs":{"total":95995035648,"available":95842553856,"internally_reserved":0,"allocated":962560,"data_stored":490071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173084,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93572068,"statfs":{"total":95995035648,"available":95817797632,"internally_reserved":0,"allocated":1536000,"data_stored":1080294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144692,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600460,"statfs":{"total":95995035648,"available":95846871040,"internally_reserved":0,"allocated":905216,"data_stored":488323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184992,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93560160,"statfs":{"total":95995035648,"available":95805603840,"internally_reserved":0,"allocated":950272,"data_stored":489112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":188283335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650390,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159748,"kb_used_data":1092,"kb_used_omap":4,"kb_used_meta":158459,"kb_avail":93585404,"statfs":{"total":95995035648,"available":95831453696,"internally_reserved":0,"allocated":1118208,"data_stored":584415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":162262691},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659223,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94732,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93650420,"statfs":{"total":95995035648,"available":95898030080,"internally_reserved":0,"allocated":929792,"data_stored":488892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:34:47.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:34:47.055 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:47.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:47.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:47.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.228 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:47.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:47.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:47.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:47.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:47.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:47.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:47.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:47.627 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:34:47.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:47.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:47.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:47.860 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:47.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:47.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:48.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.118 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:48.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:48.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:48.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:47 smithi055 ceph-mon[40587]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:34:48.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:48.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:47 smithi055 ceph-mon[29406]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:34:48.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:47 smithi067 ceph-mon[33346]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T21:34:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:48.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.377 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:48.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:48.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:48.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:48.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:48.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:48.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:48.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:48.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:48.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:48.930 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_0' removed 2024-03-26T21:34:48.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.016 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:49.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:49.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:49.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:48 smithi067 ceph-mon[33346]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:34:49.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:49.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:48 smithi055 ceph-mon[40587]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:49.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:34:49.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:48 smithi055 ceph-mon[29406]: pgmap v257: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:49.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:34:49.265 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:49.268 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:49.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:49.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:49.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:49.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:49.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:49.648 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:49.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:49.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:49.877 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:49.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:49.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:50.003 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:50.003 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:50.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:50.034 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:50.039 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:34:50.039 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:34:50.039 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:34:50.039 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:34:50.040 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:34:50.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.162 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:50.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:34:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:49 smithi067 ceph-mon[33346]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:34:50.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:34:50.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:49 smithi055 ceph-mon[40587]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:34:50.249 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1731834022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:34:50.249 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:49 smithi055 ceph-mon[29406]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T21:34:50.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:50.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:50.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.414 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:50.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:50.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.539 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:50.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:50.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:50.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:50.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:50.806 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T21:34:50.807 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T21:34:50.807 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-26T21:34:50.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-26T21:34:50.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:50.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:50.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:51.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:51.055 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:51.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:51.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:51.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:50 smithi055 ceph-mon[40587]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:51.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:50 smithi055 ceph-mon[29406]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:50 smithi067 ceph-mon[33346]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:51.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:51.310 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:51.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:51.440 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:51.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:51.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:51.697 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:51.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:51.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:51.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:51.947 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:51.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:52.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:52.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:52.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:52.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:52.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:52.332 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:52.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:52.458 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:52.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:52.627 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:34:52.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:52.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:52 smithi055 ceph-mon[40587]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:34:52.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:52.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:52 smithi055 ceph-mon[29406]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:34:52.754 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:52.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:52.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:52.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:52.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:52.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:52.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:52 smithi067 ceph-mon[33346]: pgmap v260: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:34:52.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:34:53.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:53.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:53.201 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.226 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:53.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:53.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:53.355 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:53.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:53.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:53.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:53.717 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:53.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:53.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:53.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:53.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[40587]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:34:53.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:53.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:53.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:53 smithi055 ceph-mon[29406]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:34:53.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:53.871 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:53.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:53 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:34:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2822391656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:34:53.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:53 smithi067 ceph-mon[33346]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T21:34:53.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:53.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:54.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:54.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:54.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:54.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:54.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:34:54.294 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:54.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:54.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:54.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.509 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:54.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:54.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:54.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:54.768 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:54.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:54.894 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:54.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:54.904 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:54 smithi055 ceph-mon[40587]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-26T21:34:54.904 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:54 smithi055 ceph-mon[29406]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-26T21:34:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:54 smithi067 ceph-mon[33346]: pgmap v262: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-26T21:34:54.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:55.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:55.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:55.149 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:55.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:55.275 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:55.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:55.406 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:55.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:55.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.534 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:55.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:55.635 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:55.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:55.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:55.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:34:55.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:55.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:55.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:56.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:56.051 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:56.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:56.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:56.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:56.440 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:56.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:56.605 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:34:56.625 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:34:56.645 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:34:56.666 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:34:56.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:56.687 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:34:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:56.707 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:34:56.728 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:34:56.749 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:34:56.750 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:34:56.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:56.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:56.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:56.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:56 smithi067 ceph-mon[33346]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2924418991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:56.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:34:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:34:57.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:56 smithi055 ceph-mon[40587]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2924418991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:57.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:56 smithi055 ceph-mon[29406]: pgmap v263: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:34:57.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2924418991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:34:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:57.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:57.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:57.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:57.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.341 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:57.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:57.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:34:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.471 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:57.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:34:57.572 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.597 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:57.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:57.686 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_1' created 2024-03-26T21:34:57.698 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:34:57.724 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:34:57.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:34:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.855 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:57.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1700437871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1700437871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:57.924 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:57.956 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:57.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:57.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:34:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1700437871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T21:34:58.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:58.100 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:34:58.111 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:58.213 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:58.237 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:58.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:34:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.368 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:58.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:34:58.469 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:58.538 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:34:58.540 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:34:58.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:58.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:58.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[40587]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:58.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T21:34:58.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[40587]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:34:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[29406]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T21:34:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-mon[29406]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:34:58.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.871 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:58.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:58 smithi067 ceph-mon[33346]: pgmap v264: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:34:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T21:34:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:58 smithi067 ceph-mon[33346]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T21:34:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:58.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:58.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:34:59.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.121 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:59.223 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:34:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:34:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:34:59.341 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-03-26T21:34:57.680819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1301,"num_read_kb":1147,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":667,"ondisk_log_size":667,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1149596,"kb_used_data":7932,"kb_used_omap":15,"kb_used_meta":1141552,"kb_avail":748811620,"statfs":{"total":767960285184,"available":766783098880,"internally_reserved":0,"allocated":8122368,"data_stored":5021968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1168950030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.286508"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":141,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896111+0000","last_change":"2024-03-26T21:32:55.466524+0000","last_active":"2024-03-26T21:34:52.896111+0000","last_peered":"2024-03-26T21:34:52.896111+0000","last_clean":"2024-03-26T21:34:52.896111+0000","last_became_active":"2024-03-26T21:32:55.466152+0000","last_became_peered":"2024-03-26T21:32:55.466152+0000","last_unstale":"2024-03-26T21:34:52.896111+0000","last_undegraded":"2024-03-26T21:34:52.896111+0000","last_fullsized":"2024-03-26T21:34:52.896111+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:03.491243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:03.491243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638262+0000","last_change":"2024-03-26T21:34:20.509053+0000","last_active":"2024-03-26T21:34:52.638262+0000","last_peered":"2024-03-26T21:34:52.638262+0000","last_clean":"2024-03-26T21:34:52.638262+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:34:52.638262+0000","last_undegraded":"2024-03-26T21:34:52.638262+0000","last_fullsized":"2024-03-26T21:34:52.638262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.508804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.508804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:37.144539+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895795+0000","last_change":"2024-03-26T21:32:14.926305+0000","last_active":"2024-03-26T21:34:52.895795+0000","last_peered":"2024-03-26T21:34:52.895795+0000","last_clean":"2024-03-26T21:34:52.895795+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:34:52.895795+0000","last_undegraded":"2024-03-26T21:34:52.895795+0000","last_fullsized":"2024-03-26T21:34:52.895795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:14.926185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:32:14.926185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":169,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896838+0000","last_change":"2024-03-26T21:34:20.720386+0000","last_active":"2024-03-26T21:34:52.896838+0000","last_peered":"2024-03-26T21:34:52.896838+0000","last_clean":"2024-03-26T21:34:52.896838+0000","last_became_active":"2024-03-26T21:32:55.468198+0000","last_became_peered":"2024-03-26T21:32:55.468198+0000","last_unstale":"2024-03-26T21:34:52.896838+0000","last_undegraded":"2024-03-26T21:34:52.896838+0000","last_fullsized":"2024-03-26T21:34:52.896838+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:20.720301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.720301+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:26.959221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895947+0000","last_change":"2024-03-26T21:34:24.368408+0000","last_active":"2024-03-26T21:34:52.895947+0000","last_peered":"2024-03-26T21:34:52.895947+0000","last_clean":"2024-03-26T21:34:52.895947+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:34:52.895947+0000","last_undegraded":"2024-03-26T21:34:52.895947+0000","last_fullsized":"2024-03-26T21:34:52.895947+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.368161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.690462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":149,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896240+0000","last_change":"2024-03-26T21:34:27.847069+0000","last_active":"2024-03-26T21:34:52.896240+0000","last_peered":"2024-03-26T21:34:52.896240+0000","last_clean":"2024-03-26T21:34:52.896240+0000","last_became_active":"2024-03-26T21:32:55.468736+0000","last_became_peered":"2024-03-26T21:32:55.468736+0000","last_unstale":"2024-03-26T21:34:52.896240+0000","last_undegraded":"2024-03-26T21:34:52.896240+0000","last_fullsized":"2024-03-26T21:34:52.896240+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.846942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.846942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.712678+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896068+0000","last_change":"2024-03-26T21:34:30.380605+0000","last_active":"2024-03-26T21:34:52.896068+0000","last_peered":"2024-03-26T21:34:52.896068+0000","last_clean":"2024-03-26T21:34:52.896068+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:34:52.896068+0000","last_undegraded":"2024-03-26T21:34:52.896068+0000","last_fullsized":"2024-03-26T21:34:52.896068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:30.379661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.379661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.318868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":260,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895770+0000","last_change":"2024-03-26T21:32:13.108197+0000","last_active":"2024-03-26T21:34:52.895770+0000","last_peered":"2024-03-26T21:34:52.895770+0000","last_clean":"2024-03-26T21:34:52.895770+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:34:52.895770+0000","last_undegraded":"2024-03-26T21:34:52.895770+0000","last_fullsized":"2024-03-26T21:34:52.895770+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:13.107771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.107771+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344881+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895310+0000","last_change":"2024-03-26T21:31:42.540278+0000","last_active":"2024-03-26T21:34:52.895310+0000","last_peered":"2024-03-26T21:34:52.895310+0000","last_clean":"2024-03-26T21:34:52.895310+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:34:52.895310+0000","last_undegraded":"2024-03-26T21:34:52.895310+0000","last_fullsized":"2024-03-26T21:34:52.895310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:42.540161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:42.540161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878776+0000","last_change":"2024-03-26T21:34:31.792345+0000","last_active":"2024-03-26T21:34:52.878776+0000","last_peered":"2024-03-26T21:34:52.878776+0000","last_clean":"2024-03-26T21:34:52.878776+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:34:52.878776+0000","last_undegraded":"2024-03-26T21:34:52.878776+0000","last_fullsized":"2024-03-26T21:34:52.878776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:31.792268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.792268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.823508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879013+0000","last_change":"2024-03-26T21:33:22.738629+0000","last_active":"2024-03-26T21:34:52.879013+0000","last_peered":"2024-03-26T21:34:52.879013+0000","last_clean":"2024-03-26T21:34:52.879013+0000","last_became_active":"2024-03-26T21:32:55.465246+0000","last_became_peered":"2024-03-26T21:32:55.465246+0000","last_unstale":"2024-03-26T21:34:52.879013+0000","last_undegraded":"2024-03-26T21:34:52.879013+0000","last_fullsized":"2024-03-26T21:34:52.879013+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.738510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.738510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"167'13","reported_seq":173,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895959+0000","last_change":"2024-03-26T21:34:30.923713+0000","last_active":"2024-03-26T21:34:52.895959+0000","last_peered":"2024-03-26T21:34:52.895959+0000","last_clean":"2024-03-26T21:34:52.895959+0000","last_became_active":"2024-03-26T21:32:55.480936+0000","last_became_peered":"2024-03-26T21:32:55.480936+0000","last_unstale":"2024-03-26T21:34:52.895959+0000","last_undegraded":"2024-03-26T21:34:52.895959+0000","last_fullsized":"2024-03-26T21:34:52.895959+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.923562+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:40.154880+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878473+0000","last_change":"2024-03-26T21:34:22.762001+0000","last_active":"2024-03-26T21:34:52.878473+0000","last_peered":"2024-03-26T21:34:52.878473+0000","last_clean":"2024-03-26T21:34:52.878473+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:34:52.878473+0000","last_undegraded":"2024-03-26T21:34:52.878473+0000","last_fullsized":"2024-03-26T21:34:52.878473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:22.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:34:22.761871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.252793+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878462+0000","last_change":"2024-03-26T21:34:23.724557+0000","last_active":"2024-03-26T21:34:52.878462+0000","last_peered":"2024-03-26T21:34:52.878462+0000","last_clean":"2024-03-26T21:34:52.878462+0000","last_became_active":"2024-03-26T21:32:20.904068+0000","last_became_peered":"2024-03-26T21:32:20.904068+0000","last_unstale":"2024-03-26T21:34:52.878462+0000","last_undegraded":"2024-03-26T21:34:52.878462+0000","last_fullsized":"2024-03-26T21:34:52.878462+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.724419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.724419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.040112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878920+0000","last_change":"2024-03-26T21:33:25.751247+0000","last_active":"2024-03-26T21:34:52.878920+0000","last_peered":"2024-03-26T21:34:52.878920+0000","last_clean":"2024-03-26T21:34:52.878920+0000","last_became_active":"2024-03-26T21:32:55.466491+0000","last_became_peered":"2024-03-26T21:32:55.466491+0000","last_unstale":"2024-03-26T21:34:52.878920+0000","last_undegraded":"2024-03-26T21:34:52.878920+0000","last_fullsized":"2024-03-26T21:34:52.878920+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:25.751149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:25.751149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896047+0000","last_change":"2024-03-26T21:34:11.780408+0000","last_active":"2024-03-26T21:34:52.896047+0000","last_peered":"2024-03-26T21:34:52.896047+0000","last_clean":"2024-03-26T21:34:52.896047+0000","last_became_active":"2024-03-26T21:32:55.476272+0000","last_became_peered":"2024-03-26T21:32:55.476272+0000","last_unstale":"2024-03-26T21:34:52.896047+0000","last_undegraded":"2024-03-26T21:34:52.896047+0000","last_fullsized":"2024-03-26T21:34:52.896047+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:34:11.780127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.780127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:13.819455+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.897260+0000","last_change":"2024-03-26T21:34:06.784378+0000","last_active":"2024-03-26T21:34:52.897260+0000","last_peered":"2024-03-26T21:34:52.897260+0000","last_clean":"2024-03-26T21:34:52.897260+0000","last_became_active":"2024-03-26T21:32:55.467335+0000","last_became_peered":"2024-03-26T21:32:55.467335+0000","last_unstale":"2024-03-26T21:34:52.897260+0000","last_undegraded":"2024-03-26T21:34:52.897260+0000","last_fullsized":"2024-03-26T21:34:52.897260+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:06.784228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:06.784228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:18.860098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638413+0000","last_change":"2024-03-26T21:33:18.800182+0000","last_active":"2024-03-26T21:34:52.638413+0000","last_peered":"2024-03-26T21:34:52.638413+0000","last_clean":"2024-03-26T21:34:52.638413+0000","last_became_active":"2024-03-26T21:32:55.472908+0000","last_became_peered":"2024-03-26T21:32:55.472908+0000","last_unstale":"2024-03-26T21:34:52.638413+0000","last_undegraded":"2024-03-26T21:34:52.638413+0000","last_fullsized":"2024-03-26T21:34:52.638413+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:18.799990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:18.799990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.449159+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895978+0000","last_change":"2024-03-26T21:34:27.088958+0000","last_active":"2024-03-26T21:34:52.895978+0000","last_peered":"2024-03-26T21:34:52.895978+0000","last_clean":"2024-03-26T21:34:52.895978+0000","last_became_active":"2024-03-26T21:32:55.479242+0000","last_became_peered":"2024-03-26T21:32:55.479242+0000","last_unstale":"2024-03-26T21:34:52.895978+0000","last_undegraded":"2024-03-26T21:34:52.895978+0000","last_fullsized":"2024-03-26T21:34:52.895978+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.088840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.088840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:49.279253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637242+0000","last_change":"2024-03-26T21:32:29.620716+0000","last_active":"2024-03-26T21:34:52.637242+0000","last_peered":"2024-03-26T21:34:52.637242+0000","last_clean":"2024-03-26T21:34:52.637242+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:34:52.637242+0000","last_undegraded":"2024-03-26T21:34:52.637242+0000","last_fullsized":"2024-03-26T21:34:52.637242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:32:29.620574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.620574+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":228,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896473+0000","last_change":"2024-03-26T21:32:30.360160+0000","last_active":"2024-03-26T21:34:52.896473+0000","last_peered":"2024-03-26T21:34:52.896473+0000","last_clean":"2024-03-26T21:34:52.896473+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:34:52.896473+0000","last_undegraded":"2024-03-26T21:34:52.896473+0000","last_fullsized":"2024-03-26T21:34:52.896473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.360050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.360050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":149,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896667+0000","last_change":"2024-03-26T21:34:25.862079+0000","last_active":"2024-03-26T21:34:52.896667+0000","last_peered":"2024-03-26T21:34:52.896667+0000","last_clean":"2024-03-26T21:34:52.896667+0000","last_became_active":"2024-03-26T21:32:55.465947+0000","last_became_peered":"2024-03-26T21:32:55.465947+0000","last_unstale":"2024-03-26T21:34:52.896667+0000","last_undegraded":"2024-03-26T21:34:52.896667+0000","last_fullsized":"2024-03-26T21:34:52.896667+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.861995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.861995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.431656+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":153,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896446+0000","last_change":"2024-03-26T21:32:55.477901+0000","last_active":"2024-03-26T21:34:52.896446+0000","last_peered":"2024-03-26T21:34:52.896446+0000","last_clean":"2024-03-26T21:34:52.896446+0000","last_became_active":"2024-03-26T21:32:55.477250+0000","last_became_peered":"2024-03-26T21:32:55.477250+0000","last_unstale":"2024-03-26T21:34:52.896446+0000","last_undegraded":"2024-03-26T21:34:52.896446+0000","last_fullsized":"2024-03-26T21:34:52.896446+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:05.464827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:05.464827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":259,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895715+0000","last_change":"2024-03-26T21:34:09.286797+0000","last_active":"2024-03-26T21:34:52.895715+0000","last_peered":"2024-03-26T21:34:52.895715+0000","last_clean":"2024-03-26T21:34:52.895715+0000","last_became_active":"2024-03-26T21:32:55.470311+0000","last_became_peered":"2024-03-26T21:32:55.470311+0000","last_unstale":"2024-03-26T21:34:52.895715+0000","last_undegraded":"2024-03-26T21:34:52.895715+0000","last_fullsized":"2024-03-26T21:34:52.895715+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:09.286636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.286636+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.511715+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634020+0000","last_change":"2024-03-26T21:34:25.733995+0000","last_active":"2024-03-26T21:34:52.634020+0000","last_peered":"2024-03-26T21:34:52.634020+0000","last_clean":"2024-03-26T21:34:52.634020+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:34:52.634020+0000","last_undegraded":"2024-03-26T21:34:52.634020+0000","last_fullsized":"2024-03-26T21:34:52.634020+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:34:25.733888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.733888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:27.393061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"134'10","reported_seq":339,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:54.963910+0000","last_change":"2024-03-26T21:34:27.049262+0000","last_active":"2024-03-26T21:34:54.963910+0000","last_peered":"2024-03-26T21:34:54.963910+0000","last_clean":"2024-03-26T21:34:54.963910+0000","last_became_active":"2024-03-26T21:32:55.477677+0000","last_became_peered":"2024-03-26T21:32:55.477677+0000","last_unstale":"2024-03-26T21:34:54.963910+0000","last_undegraded":"2024-03-26T21:34:54.963910+0000","last_fullsized":"2024-03-26T21:34:54.963910+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:27.048989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.048989+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.859023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":250,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638345+0000","last_change":"2024-03-26T21:33:24.786987+0000","last_active":"2024-03-26T21:34:52.638345+0000","last_peered":"2024-03-26T21:34:52.638345+0000","last_clean":"2024-03-26T21:34:52.638345+0000","last_became_active":"2024-03-26T21:32:55.469277+0000","last_became_peered":"2024-03-26T21:32:55.469277+0000","last_unstale":"2024-03-26T21:34:52.638345+0000","last_undegraded":"2024-03-26T21:34:52.638345+0000","last_fullsized":"2024-03-26T21:34:52.638345+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:33:24.786583+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:33:24.786583+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.449116+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896251+0000","last_change":"2024-03-26T21:32:22.112186+0000","last_active":"2024-03-26T21:34:52.896251+0000","last_peered":"2024-03-26T21:34:52.896251+0000","last_clean":"2024-03-26T21:34:52.896251+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:34:52.896251+0000","last_undegraded":"2024-03-26T21:34:52.896251+0000","last_fullsized":"2024-03-26T21:34:52.896251+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:32:22.112001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.112001+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638113+0000","last_change":"2024-03-26T21:33:12.751050+0000","last_active":"2024-03-26T21:34:52.638113+0000","last_peered":"2024-03-26T21:34:52.638113+0000","last_clean":"2024-03-26T21:34:52.638113+0000","last_became_active":"2024-03-26T21:32:55.465456+0000","last_became_peered":"2024-03-26T21:32:55.465456+0000","last_unstale":"2024-03-26T21:34:52.638113+0000","last_undegraded":"2024-03-26T21:34:52.638113+0000","last_fullsized":"2024-03-26T21:34:52.638113+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:12.750915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:12.750915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.449095+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896454+0000","last_change":"2024-03-26T21:31:28.707417+0000","last_active":"2024-03-26T21:34:52.896454+0000","last_peered":"2024-03-26T21:34:52.896454+0000","last_clean":"2024-03-26T21:34:52.896454+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:34:52.896454+0000","last_undegraded":"2024-03-26T21:34:52.896454+0000","last_fullsized":"2024-03-26T21:34:52.896454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:31:28.707018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895383+0000","last_change":"2024-03-26T21:31:39.883235+0000","last_active":"2024-03-26T21:34:52.895383+0000","last_peered":"2024-03-26T21:34:52.895383+0000","last_clean":"2024-03-26T21:34:52.895383+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:34:52.895383+0000","last_undegraded":"2024-03-26T21:34:52.895383+0000","last_fullsized":"2024-03-26T21:34:52.895383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:39.883105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:31:39.883105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344843+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"167'16","reported_seq":276,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895364+0000","last_change":"2024-03-26T21:34:28.014694+0000","last_active":"2024-03-26T21:34:52.895364+0000","last_peered":"2024-03-26T21:34:52.895364+0000","last_clean":"2024-03-26T21:34:52.895364+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:34:52.895364+0000","last_undegraded":"2024-03-26T21:34:52.895364+0000","last_fullsized":"2024-03-26T21:34:52.895364+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'16","last_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.014533+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.217243+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.633666+0000","last_change":"2024-03-26T21:33:22.621399+0000","last_active":"2024-03-26T21:34:52.633666+0000","last_peered":"2024-03-26T21:34:52.633666+0000","last_clean":"2024-03-26T21:34:52.633666+0000","last_became_active":"2024-03-26T21:32:55.471141+0000","last_became_peered":"2024-03-26T21:32:55.471141+0000","last_unstale":"2024-03-26T21:34:52.633666+0000","last_undegraded":"2024-03-26T21:34:52.633666+0000","last_fullsized":"2024-03-26T21:34:52.633666+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.621257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.621257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.374471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895947+0000","last_change":"2024-03-26T21:34:23.051204+0000","last_active":"2024-03-26T21:34:52.895947+0000","last_peered":"2024-03-26T21:34:52.895947+0000","last_clean":"2024-03-26T21:34:52.895947+0000","last_became_active":"2024-03-26T21:32:55.469690+0000","last_became_peered":"2024-03-26T21:32:55.469690+0000","last_unstale":"2024-03-26T21:34:52.895947+0000","last_undegraded":"2024-03-26T21:34:52.895947+0000","last_fullsized":"2024-03-26T21:34:52.895947+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.051104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.051104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.773972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895996+0000","last_change":"2024-03-26T21:32:36.887352+0000","last_active":"2024-03-26T21:34:52.895996+0000","last_peered":"2024-03-26T21:34:52.895996+0000","last_clean":"2024-03-26T21:34:52.895996+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:34:52.895996+0000","last_undegraded":"2024-03-26T21:34:52.895996+0000","last_fullsized":"2024-03-26T21:34:52.895996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:36.887160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.887160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895162+0000","last_change":"2024-03-26T21:34:25.343996+0000","last_active":"2024-03-26T21:34:52.895162+0000","last_peered":"2024-03-26T21:34:52.895162+0000","last_clean":"2024-03-26T21:34:52.895162+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:34:52.895162+0000","last_undegraded":"2024-03-26T21:34:52.895162+0000","last_fullsized":"2024-03-26T21:34:52.895162+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:34:25.343633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.343633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.287164+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896155+0000","last_change":"2024-03-26T21:32:29.393768+0000","last_active":"2024-03-26T21:34:52.896155+0000","last_peered":"2024-03-26T21:34:52.896155+0000","last_clean":"2024-03-26T21:34:52.896155+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:34:52.896155+0000","last_undegraded":"2024-03-26T21:34:52.896155+0000","last_fullsized":"2024-03-26T21:34:52.896155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:29.393601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.393601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.345012+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634043+0000","last_change":"2024-03-26T21:33:13.577559+0000","last_active":"2024-03-26T21:34:52.634043+0000","last_peered":"2024-03-26T21:34:52.634043+0000","last_clean":"2024-03-26T21:34:52.634043+0000","last_became_active":"2024-03-26T21:32:55.474210+0000","last_became_peered":"2024-03-26T21:32:55.474210+0000","last_unstale":"2024-03-26T21:34:52.634043+0000","last_undegraded":"2024-03-26T21:34:52.634043+0000","last_fullsized":"2024-03-26T21:34:52.634043+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:13.577485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:13.577485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.405270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879010+0000","last_change":"2024-03-26T21:33:30.764236+0000","last_active":"2024-03-26T21:34:52.879010+0000","last_peered":"2024-03-26T21:34:52.879010+0000","last_clean":"2024-03-26T21:34:52.879010+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:34:52.879010+0000","last_undegraded":"2024-03-26T21:34:52.879010+0000","last_fullsized":"2024-03-26T21:34:52.879010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.763924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.763924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.316480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":254,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895720+0000","last_change":"2024-03-26T21:32:15.138279+0000","last_active":"2024-03-26T21:34:52.895720+0000","last_peered":"2024-03-26T21:34:52.895720+0000","last_clean":"2024-03-26T21:34:52.895720+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:34:52.895720+0000","last_undegraded":"2024-03-26T21:34:52.895720+0000","last_fullsized":"2024-03-26T21:34:52.895720+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:32:15.138133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:15.138133+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344862+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895106+0000","last_change":"2024-03-26T21:32:26.906415+0000","last_active":"2024-03-26T21:34:52.895106+0000","last_peered":"2024-03-26T21:34:52.895106+0000","last_clean":"2024-03-26T21:34:52.895106+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:34:52.895106+0000","last_undegraded":"2024-03-26T21:34:52.895106+0000","last_fullsized":"2024-03-26T21:34:52.895106+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.906307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.906307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":231,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895854+0000","last_change":"2024-03-26T21:31:32.871165+0000","last_active":"2024-03-26T21:34:52.895854+0000","last_peered":"2024-03-26T21:34:52.895854+0000","last_clean":"2024-03-26T21:34:52.895854+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:34:52.895854+0000","last_undegraded":"2024-03-26T21:34:52.895854+0000","last_fullsized":"2024-03-26T21:34:52.895854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:32.871039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.871039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344827+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634211+0000","last_change":"2024-03-26T21:33:17.583457+0000","last_active":"2024-03-26T21:34:52.634211+0000","last_peered":"2024-03-26T21:34:52.634211+0000","last_clean":"2024-03-26T21:34:52.634211+0000","last_became_active":"2024-03-26T21:32:55.470351+0000","last_became_peered":"2024-03-26T21:32:55.470351+0000","last_unstale":"2024-03-26T21:34:52.634211+0000","last_undegraded":"2024-03-26T21:34:52.634211+0000","last_fullsized":"2024-03-26T21:34:52.634211+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:17.583338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:17.583338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.405348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638722+0000","last_change":"2024-03-26T21:34:13.533465+0000","last_active":"2024-03-26T21:34:52.638722+0000","last_peered":"2024-03-26T21:34:52.638722+0000","last_clean":"2024-03-26T21:34:52.638722+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:34:52.638722+0000","last_undegraded":"2024-03-26T21:34:52.638722+0000","last_fullsized":"2024-03-26T21:34:52.638722+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:34:13.533314+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.533314+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.625607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895384+0000","last_change":"2024-03-26T21:31:38.898103+0000","last_active":"2024-03-26T21:34:52.895384+0000","last_peered":"2024-03-26T21:34:52.895384+0000","last_clean":"2024-03-26T21:34:52.895384+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:34:52.895384+0000","last_undegraded":"2024-03-26T21:34:52.895384+0000","last_fullsized":"2024-03-26T21:34:52.895384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:38.897810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.897810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344792+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638625+0000","last_change":"2024-03-26T21:34:25.554360+0000","last_active":"2024-03-26T21:34:52.638625+0000","last_peered":"2024-03-26T21:34:52.638625+0000","last_clean":"2024-03-26T21:34:52.638625+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:34:52.638625+0000","last_undegraded":"2024-03-26T21:34:52.638625+0000","last_fullsized":"2024-03-26T21:34:52.638625+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.554230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.554230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.463566+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":156,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896485+0000","last_change":"2024-03-26T21:32:55.467765+0000","last_active":"2024-03-26T21:34:52.896485+0000","last_peered":"2024-03-26T21:34:52.896485+0000","last_clean":"2024-03-26T21:34:52.896485+0000","last_became_active":"2024-03-26T21:32:55.467375+0000","last_became_peered":"2024-03-26T21:32:55.467375+0000","last_unstale":"2024-03-26T21:34:52.896485+0000","last_undegraded":"2024-03-26T21:34:52.896485+0000","last_fullsized":"2024-03-26T21:34:52.896485+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:26.155650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:26.155650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":137,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896331+0000","last_change":"2024-03-26T21:34:10.824961+0000","last_active":"2024-03-26T21:34:52.896331+0000","last_peered":"2024-03-26T21:34:52.896331+0000","last_clean":"2024-03-26T21:34:52.896331+0000","last_became_active":"2024-03-26T21:32:55.477977+0000","last_became_peered":"2024-03-26T21:32:55.477977+0000","last_unstale":"2024-03-26T21:34:52.896331+0000","last_undegraded":"2024-03-26T21:34:52.896331+0000","last_fullsized":"2024-03-26T21:34:52.896331+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:34:10.824829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.824829+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.688410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":228,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895790+0000","last_change":"2024-03-26T21:31:40.605350+0000","last_active":"2024-03-26T21:34:52.895790+0000","last_peered":"2024-03-26T21:34:52.895790+0000","last_clean":"2024-03-26T21:34:52.895790+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:34:52.895790+0000","last_undegraded":"2024-03-26T21:34:52.895790+0000","last_fullsized":"2024-03-26T21:34:52.895790+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:31:40.605038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:31:40.605038+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":325,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879249+0000","last_change":"2024-03-26T21:34:11.674348+0000","last_active":"2024-03-26T21:34:52.879249+0000","last_peered":"2024-03-26T21:34:52.879249+0000","last_clean":"2024-03-26T21:34:52.879249+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:34:52.879249+0000","last_undegraded":"2024-03-26T21:34:52.879249+0000","last_fullsized":"2024-03-26T21:34:52.879249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:34:11.674203+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.674203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:32.191741+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.633822+0000","last_change":"2024-03-26T21:33:15.534591+0000","last_active":"2024-03-26T21:34:52.633822+0000","last_peered":"2024-03-26T21:34:52.633822+0000","last_clean":"2024-03-26T21:34:52.633822+0000","last_became_active":"2024-03-26T21:32:55.473442+0000","last_became_peered":"2024-03-26T21:32:55.473442+0000","last_unstale":"2024-03-26T21:34:52.633822+0000","last_undegraded":"2024-03-26T21:34:52.633822+0000","last_fullsized":"2024-03-26T21:34:52.633822+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:15.534472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:15.534472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.405312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"166'15","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896675+0000","last_change":"2024-03-26T21:34:27.404044+0000","last_active":"2024-03-26T21:34:52.896675+0000","last_peered":"2024-03-26T21:34:52.896675+0000","last_clean":"2024-03-26T21:34:52.896675+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:34:52.896675+0000","last_undegraded":"2024-03-26T21:34:52.896675+0000","last_fullsized":"2024-03-26T21:34:52.896675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.403785+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.739092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":144,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.897283+0000","last_change":"2024-03-26T21:32:55.466193+0000","last_active":"2024-03-26T21:34:52.897283+0000","last_peered":"2024-03-26T21:34:52.897283+0000","last_clean":"2024-03-26T21:34:52.897283+0000","last_became_active":"2024-03-26T21:32:55.465075+0000","last_became_peered":"2024-03-26T21:32:55.465075+0000","last_unstale":"2024-03-26T21:34:52.897283+0000","last_undegraded":"2024-03-26T21:34:52.897283+0000","last_fullsized":"2024-03-26T21:34:52.897283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":319,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896836+0000","last_change":"2024-03-26T21:32:36.313231+0000","last_active":"2024-03-26T21:34:52.896836+0000","last_peered":"2024-03-26T21:34:52.896836+0000","last_clean":"2024-03-26T21:34:52.896836+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:34:52.896836+0000","last_undegraded":"2024-03-26T21:34:52.896836+0000","last_fullsized":"2024-03-26T21:34:52.896836+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:36.311788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.311788+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344980+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895507+0000","last_change":"2024-03-26T21:32:23.910111+0000","last_active":"2024-03-26T21:34:52.895507+0000","last_peered":"2024-03-26T21:34:52.895507+0000","last_clean":"2024-03-26T21:34:52.895507+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:34:52.895507+0000","last_undegraded":"2024-03-26T21:34:52.895507+0000","last_fullsized":"2024-03-26T21:34:52.895507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.909981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.909981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"146'13","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879341+0000","last_change":"2024-03-26T21:33:29.776490+0000","last_active":"2024-03-26T21:34:52.879341+0000","last_peered":"2024-03-26T21:34:52.879341+0000","last_clean":"2024-03-26T21:34:52.879341+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:34:52.879341+0000","last_undegraded":"2024-03-26T21:34:52.879341+0000","last_fullsized":"2024-03-26T21:34:52.879341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.776392+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.901538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637512+0000","last_change":"2024-03-26T21:34:14.715929+0000","last_active":"2024-03-26T21:34:52.637512+0000","last_peered":"2024-03-26T21:34:52.637512+0000","last_clean":"2024-03-26T21:34:52.637512+0000","last_became_active":"2024-03-26T21:32:55.468099+0000","last_became_peered":"2024-03-26T21:32:55.468099+0000","last_unstale":"2024-03-26T21:34:52.637512+0000","last_undegraded":"2024-03-26T21:34:52.637512+0000","last_fullsized":"2024-03-26T21:34:52.637512+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.715803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.715803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.624014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":231,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896738+0000","last_change":"2024-03-26T21:32:23.286349+0000","last_active":"2024-03-26T21:34:52.896738+0000","last_peered":"2024-03-26T21:34:52.896738+0000","last_clean":"2024-03-26T21:34:52.896738+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:34:52.896738+0000","last_undegraded":"2024-03-26T21:34:52.896738+0000","last_fullsized":"2024-03-26T21:34:52.896738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.286238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.286238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344947+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.633744+0000","last_change":"2024-03-26T21:33:23.637713+0000","last_active":"2024-03-26T21:34:52.633744+0000","last_peered":"2024-03-26T21:34:52.633744+0000","last_clean":"2024-03-26T21:34:52.633744+0000","last_became_active":"2024-03-26T21:32:55.470296+0000","last_became_peered":"2024-03-26T21:32:55.470296+0000","last_unstale":"2024-03-26T21:34:52.633744+0000","last_undegraded":"2024-03-26T21:34:52.633744+0000","last_fullsized":"2024-03-26T21:34:52.633744+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:23.637591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:23.637591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.405330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896349+0000","last_change":"2024-03-26T21:31:38.685593+0000","last_active":"2024-03-26T21:34:52.896349+0000","last_peered":"2024-03-26T21:34:52.896349+0000","last_clean":"2024-03-26T21:34:52.896349+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:34:52.896349+0000","last_undegraded":"2024-03-26T21:34:52.896349+0000","last_fullsized":"2024-03-26T21:34:52.896349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:31:38.685453+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634204+0000","last_change":"2024-03-26T21:33:27.579541+0000","last_active":"2024-03-26T21:34:52.634204+0000","last_peered":"2024-03-26T21:34:52.634204+0000","last_clean":"2024-03-26T21:34:52.634204+0000","last_became_active":"2024-03-26T21:32:55.470290+0000","last_became_peered":"2024-03-26T21:32:55.470290+0000","last_unstale":"2024-03-26T21:34:52.634204+0000","last_undegraded":"2024-03-26T21:34:52.634204+0000","last_fullsized":"2024-03-26T21:34:52.634204+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:27.579465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:33:27.579465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.584258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879205+0000","last_change":"2024-03-26T21:32:41.663300+0000","last_active":"2024-03-26T21:34:52.879205+0000","last_peered":"2024-03-26T21:34:52.879205+0000","last_clean":"2024-03-26T21:34:52.879205+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:34:52.879205+0000","last_undegraded":"2024-03-26T21:34:52.879205+0000","last_fullsized":"2024-03-26T21:34:52.879205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:41.663183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:41.663183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899918+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895419+0000","last_change":"2024-03-26T21:33:30.266914+0000","last_active":"2024-03-26T21:34:52.895419+0000","last_peered":"2024-03-26T21:34:52.895419+0000","last_clean":"2024-03-26T21:34:52.895419+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:34:52.895419+0000","last_undegraded":"2024-03-26T21:34:52.895419+0000","last_fullsized":"2024-03-26T21:34:52.895419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.266833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.266833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":258,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896666+0000","last_change":"2024-03-26T21:32:18.220547+0000","last_active":"2024-03-26T21:34:52.896666+0000","last_peered":"2024-03-26T21:34:52.896666+0000","last_clean":"2024-03-26T21:34:52.896666+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:34:52.896666+0000","last_undegraded":"2024-03-26T21:34:52.896666+0000","last_fullsized":"2024-03-26T21:34:52.896666+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:32:18.214216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.214216+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344963+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878890+0000","last_change":"2024-03-26T21:32:30.548012+0000","last_active":"2024-03-26T21:34:52.878890+0000","last_peered":"2024-03-26T21:34:52.878890+0000","last_clean":"2024-03-26T21:34:52.878890+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:34:52.878890+0000","last_undegraded":"2024-03-26T21:34:52.878890+0000","last_fullsized":"2024-03-26T21:34:52.878890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.547900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":266,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634583+0000","last_change":"2024-03-26T21:34:28.712452+0000","last_active":"2024-03-26T21:34:52.634583+0000","last_peered":"2024-03-26T21:34:52.634583+0000","last_clean":"2024-03-26T21:34:52.634583+0000","last_became_active":"2024-03-26T21:32:55.475860+0000","last_became_peered":"2024-03-26T21:32:55.475860+0000","last_unstale":"2024-03-26T21:34:52.634583+0000","last_undegraded":"2024-03-26T21:34:52.634583+0000","last_fullsized":"2024-03-26T21:34:52.634583+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:34:28.712368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.712368+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.654006+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878488+0000","last_change":"2024-03-26T21:34:32.761556+0000","last_active":"2024-03-26T21:34:52.878488+0000","last_peered":"2024-03-26T21:34:52.878488+0000","last_clean":"2024-03-26T21:34:52.878488+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:34:52.878488+0000","last_undegraded":"2024-03-26T21:34:52.878488+0000","last_fullsized":"2024-03-26T21:34:52.878488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:32.761425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.761425+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.953919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"134'10","reported_seq":327,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634076+0000","last_change":"2024-03-26T21:34:30.686544+0000","last_active":"2024-03-26T21:34:52.634076+0000","last_peered":"2024-03-26T21:34:52.634076+0000","last_clean":"2024-03-26T21:34:52.634076+0000","last_became_active":"2024-03-26T21:32:55.471127+0000","last_became_peered":"2024-03-26T21:32:55.471127+0000","last_unstale":"2024-03-26T21:34:52.634076+0000","last_undegraded":"2024-03-26T21:34:52.634076+0000","last_fullsized":"2024-03-26T21:34:52.634076+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:30.686411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.686411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.859040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896050+0000","last_change":"2024-03-26T21:32:42.225753+0000","last_active":"2024-03-26T21:34:52.896050+0000","last_peered":"2024-03-26T21:34:52.896050+0000","last_clean":"2024-03-26T21:34:52.896050+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:34:52.896050+0000","last_undegraded":"2024-03-26T21:34:52.896050+0000","last_fullsized":"2024-03-26T21:34:52.896050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:42.225645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:42.225645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344997+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634427+0000","last_change":"2024-03-26T21:33:16.562187+0000","last_active":"2024-03-26T21:34:52.634427+0000","last_peered":"2024-03-26T21:34:52.634427+0000","last_clean":"2024-03-26T21:34:52.634427+0000","last_became_active":"2024-03-26T21:32:55.471766+0000","last_became_peered":"2024-03-26T21:32:55.471766+0000","last_unstale":"2024-03-26T21:34:52.634427+0000","last_undegraded":"2024-03-26T21:34:52.634427+0000","last_fullsized":"2024-03-26T21:34:52.634427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:33:16.562082+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:33:16.562082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.405294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896790+0000","last_change":"2024-03-26T21:34:18.762352+0000","last_active":"2024-03-26T21:34:52.896790+0000","last_peered":"2024-03-26T21:34:52.896790+0000","last_clean":"2024-03-26T21:34:52.896790+0000","last_became_active":"2024-03-26T21:32:55.467883+0000","last_became_peered":"2024-03-26T21:32:55.467883+0000","last_unstale":"2024-03-26T21:34:52.896790+0000","last_undegraded":"2024-03-26T21:34:52.896790+0000","last_fullsized":"2024-03-26T21:34:52.896790+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:18.762259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.762259+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.929171+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.636961+0000","last_change":"2024-03-26T21:34:20.747940+0000","last_active":"2024-03-26T21:34:52.636961+0000","last_peered":"2024-03-26T21:34:52.636961+0000","last_clean":"2024-03-26T21:34:52.636961+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:34:52.636961+0000","last_undegraded":"2024-03-26T21:34:52.636961+0000","last_fullsized":"2024-03-26T21:34:52.636961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.747694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.747694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.007716+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.633487+0000","last_change":"2024-03-26T21:34:08.816123+0000","last_active":"2024-03-26T21:34:52.633487+0000","last_peered":"2024-03-26T21:34:52.633487+0000","last_clean":"2024-03-26T21:34:52.633487+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:34:52.633487+0000","last_undegraded":"2024-03-26T21:34:52.633487+0000","last_fullsized":"2024-03-26T21:34:52.633487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.816046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.816046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.663133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637330+0000","last_change":"2024-03-26T21:34:07.585143+0000","last_active":"2024-03-26T21:34:52.637330+0000","last_peered":"2024-03-26T21:34:52.637330+0000","last_clean":"2024-03-26T21:34:52.637330+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:34:52.637330+0000","last_undegraded":"2024-03-26T21:34:52.637330+0000","last_fullsized":"2024-03-26T21:34:52.637330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.585039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.585039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.578417+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634460+0000","last_change":"2024-03-26T21:34:26.757750+0000","last_active":"2024-03-26T21:34:52.634460+0000","last_peered":"2024-03-26T21:34:52.634460+0000","last_clean":"2024-03-26T21:34:52.634460+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:34:52.634460+0000","last_undegraded":"2024-03-26T21:34:52.634460+0000","last_fullsized":"2024-03-26T21:34:52.634460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.757669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.757669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:56.675099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":283,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896176+0000","last_change":"2024-03-26T21:33:25.213686+0000","last_active":"2024-03-26T21:34:52.896176+0000","last_peered":"2024-03-26T21:34:52.896176+0000","last_clean":"2024-03-26T21:34:52.896176+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:34:52.896176+0000","last_undegraded":"2024-03-26T21:34:52.896176+0000","last_fullsized":"2024-03-26T21:34:52.896176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:33:25.213564+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.080235+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895217+0000","last_change":"2024-03-26T21:31:37.722079+0000","last_active":"2024-03-26T21:34:52.895217+0000","last_peered":"2024-03-26T21:34:52.895217+0000","last_clean":"2024-03-26T21:34:52.895217+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:34:52.895217+0000","last_undegraded":"2024-03-26T21:34:52.895217+0000","last_fullsized":"2024-03-26T21:34:52.895217+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:31:37.721821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.636936+0000","last_change":"2024-03-26T21:33:29.574051+0000","last_active":"2024-03-26T21:34:52.636936+0000","last_peered":"2024-03-26T21:34:52.636936+0000","last_clean":"2024-03-26T21:34:52.636936+0000","last_became_active":"2024-03-26T21:32:55.471707+0000","last_became_peered":"2024-03-26T21:32:55.471707+0000","last_unstale":"2024-03-26T21:34:52.636936+0000","last_undegraded":"2024-03-26T21:34:52.636936+0000","last_fullsized":"2024-03-26T21:34:52.636936+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:29.573915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.573915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878927+0000","last_change":"2024-03-26T21:34:14.695183+0000","last_active":"2024-03-26T21:34:52.878927+0000","last_peered":"2024-03-26T21:34:52.878927+0000","last_clean":"2024-03-26T21:34:52.878927+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:34:52.878927+0000","last_undegraded":"2024-03-26T21:34:52.878927+0000","last_fullsized":"2024-03-26T21:34:52.878927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.695081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.695081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:17.363369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895871+0000","last_change":"2024-03-26T21:34:15.948871+0000","last_active":"2024-03-26T21:34:52.895871+0000","last_peered":"2024-03-26T21:34:52.895871+0000","last_clean":"2024-03-26T21:34:52.895871+0000","last_became_active":"2024-03-26T21:32:55.478663+0000","last_became_peered":"2024-03-26T21:32:55.478663+0000","last_unstale":"2024-03-26T21:34:52.895871+0000","last_undegraded":"2024-03-26T21:34:52.895871+0000","last_fullsized":"2024-03-26T21:34:52.895871+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:15.948730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.948730+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:17.502102+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896384+0000","last_change":"2024-03-26T21:34:10.292343+0000","last_active":"2024-03-26T21:34:52.896384+0000","last_peered":"2024-03-26T21:34:52.896384+0000","last_clean":"2024-03-26T21:34:52.896384+0000","last_became_active":"2024-03-26T21:32:55.468927+0000","last_became_peered":"2024-03-26T21:32:55.468927+0000","last_unstale":"2024-03-26T21:34:52.896384+0000","last_undegraded":"2024-03-26T21:34:52.896384+0000","last_fullsized":"2024-03-26T21:34:52.896384+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:10.292229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.292229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:14.024010+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":257,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878729+0000","last_change":"2024-03-26T21:34:16.662556+0000","last_active":"2024-03-26T21:34:52.878729+0000","last_peered":"2024-03-26T21:34:52.878729+0000","last_clean":"2024-03-26T21:34:52.878729+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:34:52.878729+0000","last_undegraded":"2024-03-26T21:34:52.878729+0000","last_fullsized":"2024-03-26T21:34:52.878729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:34:16.662471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:16.662471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.912431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":155,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895955+0000","last_change":"2024-03-26T21:32:55.480927+0000","last_active":"2024-03-26T21:34:52.895955+0000","last_peered":"2024-03-26T21:34:52.895955+0000","last_clean":"2024-03-26T21:34:52.895955+0000","last_became_active":"2024-03-26T21:32:55.480439+0000","last_became_peered":"2024-03-26T21:32:55.480439+0000","last_unstale":"2024-03-26T21:34:52.895955+0000","last_undegraded":"2024-03-26T21:34:52.895955+0000","last_fullsized":"2024-03-26T21:34:52.895955+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:15.290476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:30:15.290476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.969984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":322,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878756+0000","last_change":"2024-03-26T21:34:13.674583+0000","last_active":"2024-03-26T21:34:52.878756+0000","last_peered":"2024-03-26T21:34:52.878756+0000","last_clean":"2024-03-26T21:34:52.878756+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:34:52.878756+0000","last_undegraded":"2024-03-26T21:34:52.878756+0000","last_fullsized":"2024-03-26T21:34:52.878756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:34:13.674406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.674406+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:16.518466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896076+0000","last_change":"2024-03-26T21:34:18.955506+0000","last_active":"2024-03-26T21:34:52.896076+0000","last_peered":"2024-03-26T21:34:52.896076+0000","last_clean":"2024-03-26T21:34:52.896076+0000","last_became_active":"2024-03-26T21:32:55.478576+0000","last_became_peered":"2024-03-26T21:32:55.478576+0000","last_unstale":"2024-03-26T21:34:52.896076+0000","last_undegraded":"2024-03-26T21:34:52.896076+0000","last_fullsized":"2024-03-26T21:34:52.896076+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:18.955400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.955400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.675127+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634612+0000","last_change":"2024-03-26T21:33:28.620552+0000","last_active":"2024-03-26T21:34:52.634612+0000","last_peered":"2024-03-26T21:34:52.634612+0000","last_clean":"2024-03-26T21:34:52.634612+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:34:52.634612+0000","last_undegraded":"2024-03-26T21:34:52.634612+0000","last_fullsized":"2024-03-26T21:34:52.634612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:28.620444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:33:28.620444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:00.481746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637836+0000","last_change":"2024-03-26T21:34:24.599907+0000","last_active":"2024-03-26T21:34:52.637836+0000","last_peered":"2024-03-26T21:34:52.637836+0000","last_clean":"2024-03-26T21:34:52.637836+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:34:52.637836+0000","last_undegraded":"2024-03-26T21:34:52.637836+0000","last_fullsized":"2024-03-26T21:34:52.637836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.599655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.967423+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637020+0000","last_change":"2024-03-26T21:34:29.929261+0000","last_active":"2024-03-26T21:34:52.637020+0000","last_peered":"2024-03-26T21:34:52.637020+0000","last_clean":"2024-03-26T21:34:52.637020+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:34:52.637020+0000","last_undegraded":"2024-03-26T21:34:52.637020+0000","last_fullsized":"2024-03-26T21:34:52.637020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:34:29.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.201699+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":275,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638797+0000","last_change":"2024-03-26T21:34:10.523681+0000","last_active":"2024-03-26T21:34:52.638797+0000","last_peered":"2024-03-26T21:34:52.638797+0000","last_clean":"2024-03-26T21:34:52.638797+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:34:52.638797+0000","last_undegraded":"2024-03-26T21:34:52.638797+0000","last_fullsized":"2024-03-26T21:34:52.638797+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:34:10.523547+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.523547+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.692847+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637682+0000","last_change":"2024-03-26T21:34:32.868281+0000","last_active":"2024-03-26T21:34:52.637682+0000","last_peered":"2024-03-26T21:34:52.637682+0000","last_clean":"2024-03-26T21:34:52.637682+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:34:52.637682+0000","last_undegraded":"2024-03-26T21:34:52.637682+0000","last_fullsized":"2024-03-26T21:34:52.637682+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.868200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.530052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879264+0000","last_change":"2024-03-26T21:33:26.736579+0000","last_active":"2024-03-26T21:34:52.879264+0000","last_peered":"2024-03-26T21:34:52.879264+0000","last_clean":"2024-03-26T21:34:52.879264+0000","last_became_active":"2024-03-26T21:32:55.474113+0000","last_became_peered":"2024-03-26T21:32:55.474113+0000","last_unstale":"2024-03-26T21:34:52.879264+0000","last_undegraded":"2024-03-26T21:34:52.879264+0000","last_fullsized":"2024-03-26T21:34:52.879264+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:26.736456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:26.736456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":500,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.633628+0000","last_change":"2024-03-26T21:34:17.683647+0000","last_active":"2024-03-26T21:34:52.633628+0000","last_peered":"2024-03-26T21:34:52.633628+0000","last_clean":"2024-03-26T21:34:52.633628+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:34:52.633628+0000","last_undegraded":"2024-03-26T21:34:52.633628+0000","last_fullsized":"2024-03-26T21:34:52.633628+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:34:17.683557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:34:17.683557+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.639537+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896361+0000","last_change":"2024-03-26T21:34:28.378904+0000","last_active":"2024-03-26T21:34:52.896361+0000","last_peered":"2024-03-26T21:34:52.896361+0000","last_clean":"2024-03-26T21:34:52.896361+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:34:52.896361+0000","last_undegraded":"2024-03-26T21:34:52.896361+0000","last_fullsized":"2024-03-26T21:34:52.896361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:28.378787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.378787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:34.024112+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"134'26","reported_seq":362,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896490+0000","last_change":"2024-03-26T21:34:31.976293+0000","last_active":"2024-03-26T21:34:52.896490+0000","last_peered":"2024-03-26T21:34:52.896490+0000","last_clean":"2024-03-26T21:34:52.896490+0000","last_became_active":"2024-03-26T21:32:55.477984+0000","last_became_peered":"2024-03-26T21:32:55.477984+0000","last_unstale":"2024-03-26T21:34:52.896490+0000","last_undegraded":"2024-03-26T21:34:52.896490+0000","last_fullsized":"2024-03-26T21:34:52.896490+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'26","last_scrub_stamp":"2024-03-26T21:34:31.976190+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.976190+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.446068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634222+0000","last_change":"2024-03-26T21:34:20.660945+0000","last_active":"2024-03-26T21:34:52.634222+0000","last_peered":"2024-03-26T21:34:52.634222+0000","last_clean":"2024-03-26T21:34:52.634222+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:34:52.634222+0000","last_undegraded":"2024-03-26T21:34:52.634222+0000","last_fullsized":"2024-03-26T21:34:52.634222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.660825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.660825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.066534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638559+0000","last_change":"2024-03-26T21:34:14.558500+0000","last_active":"2024-03-26T21:34:52.638559+0000","last_peered":"2024-03-26T21:34:52.638559+0000","last_clean":"2024-03-26T21:34:52.638559+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:34:52.638559+0000","last_undegraded":"2024-03-26T21:34:52.638559+0000","last_fullsized":"2024-03-26T21:34:52.638559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.557934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.557934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.351356+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895503+0000","last_change":"2024-03-26T21:34:25.034122+0000","last_active":"2024-03-26T21:34:52.895503+0000","last_peered":"2024-03-26T21:34:52.895503+0000","last_clean":"2024-03-26T21:34:52.895503+0000","last_became_active":"2024-03-26T21:32:55.475773+0000","last_became_peered":"2024-03-26T21:32:55.475773+0000","last_unstale":"2024-03-26T21:34:52.895503+0000","last_undegraded":"2024-03-26T21:34:52.895503+0000","last_fullsized":"2024-03-26T21:34:52.895503+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.033987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.033987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:25.283140+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":633,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:54.843782+0000","last_change":"2024-03-26T21:33:29.586212+0000","last_active":"2024-03-26T21:34:54.843782+0000","last_peered":"2024-03-26T21:34:54.843782+0000","last_clean":"2024-03-26T21:34:54.843782+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:34:54.843782+0000","last_undegraded":"2024-03-26T21:34:54.843782+0000","last_fullsized":"2024-03-26T21:34:54.843782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:33:29.586133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:29.586133+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.990517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":281,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895897+0000","last_change":"2024-03-26T21:32:39.859396+0000","last_active":"2024-03-26T21:34:52.895897+0000","last_peered":"2024-03-26T21:34:52.895897+0000","last_clean":"2024-03-26T21:34:52.895897+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:34:52.895897+0000","last_undegraded":"2024-03-26T21:34:52.895897+0000","last_fullsized":"2024-03-26T21:34:52.895897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:32:39.858918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:39.858918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879440+0000","last_change":"2024-03-26T21:34:09.662284+0000","last_active":"2024-03-26T21:34:52.879440+0000","last_peered":"2024-03-26T21:34:52.879440+0000","last_clean":"2024-03-26T21:34:52.879440+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:34:52.879440+0000","last_undegraded":"2024-03-26T21:34:52.879440+0000","last_fullsized":"2024-03-26T21:34:52.879440+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.662190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.662190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.442388+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638293+0000","last_change":"2024-03-26T21:34:08.527352+0000","last_active":"2024-03-26T21:34:52.638293+0000","last_peered":"2024-03-26T21:34:52.638293+0000","last_clean":"2024-03-26T21:34:52.638293+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:34:52.638293+0000","last_undegraded":"2024-03-26T21:34:52.638293+0000","last_fullsized":"2024-03-26T21:34:52.638293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.527230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.527230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.546653+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638826+0000","last_change":"2024-03-26T21:34:09.554532+0000","last_active":"2024-03-26T21:34:52.638826+0000","last_peered":"2024-03-26T21:34:52.638826+0000","last_clean":"2024-03-26T21:34:52.638826+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:34:52.638826+0000","last_undegraded":"2024-03-26T21:34:52.638826+0000","last_fullsized":"2024-03-26T21:34:52.638826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.554099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.554099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:12.804063+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":272,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895710+0000","last_change":"2024-03-26T21:31:32.754170+0000","last_active":"2024-03-26T21:34:52.895710+0000","last_peered":"2024-03-26T21:34:52.895710+0000","last_clean":"2024-03-26T21:34:52.895710+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:34:52.895710+0000","last_undegraded":"2024-03-26T21:34:52.895710+0000","last_fullsized":"2024-03-26T21:34:52.895710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:31:32.754010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:31:32.754010+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896363+0000","last_change":"2024-03-26T21:33:24.229284+0000","last_active":"2024-03-26T21:34:52.896363+0000","last_peered":"2024-03-26T21:34:52.896363+0000","last_clean":"2024-03-26T21:34:52.896363+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:34:52.896363+0000","last_undegraded":"2024-03-26T21:34:52.896363+0000","last_fullsized":"2024-03-26T21:34:52.896363+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:33:24.229192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:59.014256+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":228,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895132+0000","last_change":"2024-03-26T21:32:13.010288+0000","last_active":"2024-03-26T21:34:52.895132+0000","last_peered":"2024-03-26T21:34:52.895132+0000","last_clean":"2024-03-26T21:34:52.895132+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:34:52.895132+0000","last_undegraded":"2024-03-26T21:34:52.895132+0000","last_fullsized":"2024-03-26T21:34:52.895132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:13.010170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.010170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":248,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879196+0000","last_change":"2024-03-26T21:32:35.525387+0000","last_active":"2024-03-26T21:34:52.879196+0000","last_peered":"2024-03-26T21:34:52.879196+0000","last_clean":"2024-03-26T21:34:52.879196+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:34:52.879196+0000","last_undegraded":"2024-03-26T21:34:52.879196+0000","last_fullsized":"2024-03-26T21:34:52.879196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:32:35.525298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.899900+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":260,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638697+0000","last_change":"2024-03-26T21:33:19.746237+0000","last_active":"2024-03-26T21:34:52.638697+0000","last_peered":"2024-03-26T21:34:52.638697+0000","last_clean":"2024-03-26T21:34:52.638697+0000","last_became_active":"2024-03-26T21:32:55.465412+0000","last_became_peered":"2024-03-26T21:32:55.465412+0000","last_unstale":"2024-03-26T21:34:52.638697+0000","last_undegraded":"2024-03-26T21:34:52.638697+0000","last_fullsized":"2024-03-26T21:34:52.638697+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:33:19.745702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:33:19.745702+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:56.449135+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":317,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:54.963705+0000","last_change":"2024-03-26T21:32:11.923454+0000","last_active":"2024-03-26T21:34:54.963705+0000","last_peered":"2024-03-26T21:34:54.963705+0000","last_clean":"2024-03-26T21:34:54.963705+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:34:54.963705+0000","last_undegraded":"2024-03-26T21:34:54.963705+0000","last_fullsized":"2024-03-26T21:34:54.963705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:11.916784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:11.916784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:58.179464+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.634011+0000","last_change":"2024-03-26T21:33:30.581306+0000","last_active":"2024-03-26T21:34:52.634011+0000","last_peered":"2024-03-26T21:34:52.634011+0000","last_clean":"2024-03-26T21:34:52.634011+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:34:52.634011+0000","last_undegraded":"2024-03-26T21:34:52.634011+0000","last_fullsized":"2024-03-26T21:34:52.634011+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.581234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.581234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895992+0000","last_change":"2024-03-26T21:34:10.057922+0000","last_active":"2024-03-26T21:34:52.895992+0000","last_peered":"2024-03-26T21:34:52.895992+0000","last_clean":"2024-03-26T21:34:52.895992+0000","last_became_active":"2024-03-26T21:32:55.479274+0000","last_became_peered":"2024-03-26T21:32:55.479274+0000","last_unstale":"2024-03-26T21:34:52.895992+0000","last_undegraded":"2024-03-26T21:34:52.895992+0000","last_fullsized":"2024-03-26T21:34:52.895992+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:34:10.057797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.057797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:23.442880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":172,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896609+0000","last_change":"2024-03-26T21:32:55.472003+0000","last_active":"2024-03-26T21:34:52.896609+0000","last_peered":"2024-03-26T21:34:52.896609+0000","last_clean":"2024-03-26T21:34:52.896609+0000","last_became_active":"2024-03-26T21:32:55.469472+0000","last_became_peered":"2024-03-26T21:32:55.469472+0000","last_unstale":"2024-03-26T21:34:52.896609+0000","last_undegraded":"2024-03-26T21:34:52.896609+0000","last_fullsized":"2024-03-26T21:34:52.896609+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:30:09.414714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:09.414714+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"134'10","reported_seq":328,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896276+0000","last_change":"2024-03-26T21:34:12.341709+0000","last_active":"2024-03-26T21:34:52.896276+0000","last_peered":"2024-03-26T21:34:52.896276+0000","last_clean":"2024-03-26T21:34:52.896276+0000","last_became_active":"2024-03-26T21:32:55.476581+0000","last_became_peered":"2024-03-26T21:32:55.476581+0000","last_unstale":"2024-03-26T21:34:52.896276+0000","last_undegraded":"2024-03-26T21:34:52.896276+0000","last_fullsized":"2024-03-26T21:34:52.896276+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:12.341310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.341310+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.217651+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":145,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896313+0000","last_change":"2024-03-26T21:34:33.905393+0000","last_active":"2024-03-26T21:34:52.896313+0000","last_peered":"2024-03-26T21:34:52.896313+0000","last_clean":"2024-03-26T21:34:52.896313+0000","last_became_active":"2024-03-26T21:32:55.476227+0000","last_became_peered":"2024-03-26T21:32:55.476227+0000","last_unstale":"2024-03-26T21:34:52.896313+0000","last_undegraded":"2024-03-26T21:34:52.896313+0000","last_fullsized":"2024-03-26T21:34:52.896313+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.905276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.905276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.967842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896560+0000","last_change":"2024-03-26T21:32:17.198343+0000","last_active":"2024-03-26T21:34:52.896560+0000","last_peered":"2024-03-26T21:34:52.896560+0000","last_clean":"2024-03-26T21:34:52.896560+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:34:52.896560+0000","last_undegraded":"2024-03-26T21:34:52.896560+0000","last_fullsized":"2024-03-26T21:34:52.896560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:17.198228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:17.198228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344914+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896115+0000","last_change":"2024-03-26T21:32:33.337863+0000","last_active":"2024-03-26T21:34:52.896115+0000","last_peered":"2024-03-26T21:34:52.896115+0000","last_clean":"2024-03-26T21:34:52.896115+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:34:52.896115+0000","last_undegraded":"2024-03-26T21:34:52.896115+0000","last_fullsized":"2024-03-26T21:34:52.896115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:32:33.337581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:33.337581+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":106,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896601+0000","last_change":"2024-03-26T21:34:07.791992+0000","last_active":"2024-03-26T21:34:52.896601+0000","last_peered":"2024-03-26T21:34:52.896601+0000","last_clean":"2024-03-26T21:34:52.896601+0000","last_became_active":"2024-03-26T21:32:55.468331+0000","last_became_peered":"2024-03-26T21:32:55.468331+0000","last_unstale":"2024-03-26T21:34:52.896601+0000","last_undegraded":"2024-03-26T21:34:52.896601+0000","last_fullsized":"2024-03-26T21:34:52.896601+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.791763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.791763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.911000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.878959+0000","last_change":"2024-03-26T21:34:15.695258+0000","last_active":"2024-03-26T21:34:52.878959+0000","last_peered":"2024-03-26T21:34:52.878959+0000","last_clean":"2024-03-26T21:34:52.878959+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:34:52.878959+0000","last_undegraded":"2024-03-26T21:34:52.878959+0000","last_fullsized":"2024-03-26T21:34:52.878959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:15.695126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.695126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.175611+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"134'10","reported_seq":199,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.897389+0000","last_change":"2024-03-26T21:32:55.480444+0000","last_active":"2024-03-26T21:34:52.897389+0000","last_peered":"2024-03-26T21:34:52.897389+0000","last_clean":"2024-03-26T21:34:52.897389+0000","last_became_active":"2024-03-26T21:32:55.478624+0000","last_became_peered":"2024-03-26T21:32:55.478624+0000","last_unstale":"2024-03-26T21:34:52.897389+0000","last_undegraded":"2024-03-26T21:34:52.897389+0000","last_fullsized":"2024-03-26T21:34:52.897389+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:30:11.407216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:30:11.407216+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.969953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896025+0000","last_change":"2024-03-26T21:31:29.955815+0000","last_active":"2024-03-26T21:34:52.896025+0000","last_peered":"2024-03-26T21:34:52.896025+0000","last_clean":"2024-03-26T21:34:52.896025+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:34:52.896025+0000","last_undegraded":"2024-03-26T21:34:52.896025+0000","last_fullsized":"2024-03-26T21:34:52.896025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:31:29.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:55.344810+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":277,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638916+0000","last_change":"2024-03-26T21:34:33.587083+0000","last_active":"2024-03-26T21:34:52.638916+0000","last_peered":"2024-03-26T21:34:52.638916+0000","last_clean":"2024-03-26T21:34:52.638916+0000","last_became_active":"2024-03-26T21:32:55.471751+0000","last_became_peered":"2024-03-26T21:32:55.471751+0000","last_unstale":"2024-03-26T21:34:52.638916+0000","last_undegraded":"2024-03-26T21:34:52.638916+0000","last_fullsized":"2024-03-26T21:34:52.638916+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:33.586983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.586983+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.630361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896337+0000","last_change":"2024-03-26T21:34:32.360553+0000","last_active":"2024-03-26T21:34:52.896337+0000","last_peered":"2024-03-26T21:34:52.896337+0000","last_clean":"2024-03-26T21:34:52.896337+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:34:52.896337+0000","last_undegraded":"2024-03-26T21:34:52.896337+0000","last_fullsized":"2024-03-26T21:34:52.896337+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.360471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.360471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.149069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.879443+0000","last_change":"2024-03-26T21:33:17.842797+0000","last_active":"2024-03-26T21:34:52.879443+0000","last_peered":"2024-03-26T21:34:52.879443+0000","last_clean":"2024-03-26T21:34:52.879443+0000","last_became_active":"2024-03-26T21:32:55.465348+0000","last_became_peered":"2024-03-26T21:32:55.465348+0000","last_unstale":"2024-03-26T21:34:52.879443+0000","last_undegraded":"2024-03-26T21:34:52.879443+0000","last_fullsized":"2024-03-26T21:34:52.879443+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:17.842709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:33:17.842709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.901514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":157,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895793+0000","last_change":"2024-03-26T21:34:24.836203+0000","last_active":"2024-03-26T21:34:52.895793+0000","last_peered":"2024-03-26T21:34:52.895793+0000","last_clean":"2024-03-26T21:34:52.895793+0000","last_became_active":"2024-03-26T21:32:55.463332+0000","last_became_peered":"2024-03-26T21:32:55.463332+0000","last_unstale":"2024-03-26T21:34:52.895793+0000","last_undegraded":"2024-03-26T21:34:52.895793+0000","last_fullsized":"2024-03-26T21:34:52.895793+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.835878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.835878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.538447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":168,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896809+0000","last_change":"2024-03-26T21:32:55.472726+0000","last_active":"2024-03-26T21:34:52.896809+0000","last_peered":"2024-03-26T21:34:52.896809+0000","last_clean":"2024-03-26T21:34:52.896809+0000","last_became_active":"2024-03-26T21:32:55.471887+0000","last_became_peered":"2024-03-26T21:32:55.471887+0000","last_unstale":"2024-03-26T21:34:52.896809+0000","last_undegraded":"2024-03-26T21:34:52.896809+0000","last_fullsized":"2024-03-26T21:34:52.896809+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:30:01.544755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:01.544755+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.637457+0000","last_change":"2024-03-26T21:34:12.660667+0000","last_active":"2024-03-26T21:34:52.637457+0000","last_peered":"2024-03-26T21:34:52.637457+0000","last_clean":"2024-03-26T21:34:52.637457+0000","last_became_active":"2024-03-26T21:32:55.470413+0000","last_became_peered":"2024-03-26T21:32:55.470413+0000","last_unstale":"2024-03-26T21:34:52.637457+0000","last_undegraded":"2024-03-26T21:34:52.637457+0000","last_fullsized":"2024-03-26T21:34:52.637457+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:12.660555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.660555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.356026+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638911+0000","last_change":"2024-03-26T21:34:26.592280+0000","last_active":"2024-03-26T21:34:52.638911+0000","last_peered":"2024-03-26T21:34:52.638911+0000","last_clean":"2024-03-26T21:34:52.638911+0000","last_became_active":"2024-03-26T21:32:55.472013+0000","last_became_peered":"2024-03-26T21:32:55.472013+0000","last_unstale":"2024-03-26T21:34:52.638911+0000","last_undegraded":"2024-03-26T21:34:52.638911+0000","last_fullsized":"2024-03-26T21:34:52.638911+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.591961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.591961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.616850+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896424+0000","last_change":"2024-03-26T21:34:33.006206+0000","last_active":"2024-03-26T21:34:52.896424+0000","last_peered":"2024-03-26T21:34:52.896424+0000","last_clean":"2024-03-26T21:34:52.896424+0000","last_became_active":"2024-03-26T21:32:55.469205+0000","last_became_peered":"2024-03-26T21:32:55.469205+0000","last_unstale":"2024-03-26T21:34:52.896424+0000","last_undegraded":"2024-03-26T21:34:52.896424+0000","last_fullsized":"2024-03-26T21:34:52.896424+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.006101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.004115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":180,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896591+0000","last_change":"2024-03-26T21:32:55.464891+0000","last_active":"2024-03-26T21:34:52.896591+0000","last_peered":"2024-03-26T21:34:52.896591+0000","last_clean":"2024-03-26T21:34:52.896591+0000","last_became_active":"2024-03-26T21:32:55.464394+0000","last_became_peered":"2024-03-26T21:32:55.464394+0000","last_unstale":"2024-03-26T21:34:52.896591+0000","last_undegraded":"2024-03-26T21:34:52.896591+0000","last_fullsized":"2024-03-26T21:34:52.896591+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:30:19.267430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:30:19.267430+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T21:34:57.654033+0000","last_change":"2024-03-26T21:34:57.654033+0000","last_active":"2024-03-26T21:34:57.654033+0000","last_peered":"2024-03-26T21:34:57.654033+0000","last_clean":"2024-03-26T21:34:57.654033+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T21:34:57.654033+0000","last_undegraded":"2024-03-26T21:34:57.654033+0000","last_fullsized":"2024-03-26T21:34:57.654033+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":141,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.897024+0000","last_change":"2024-03-26T21:32:55.472265+0000","last_active":"2024-03-26T21:34:52.897024+0000","last_peered":"2024-03-26T21:34:52.897024+0000","last_clean":"2024-03-26T21:34:52.897024+0000","last_became_active":"2024-03-26T21:32:55.471347+0000","last_became_peered":"2024-03-26T21:32:55.471347+0000","last_unstale":"2024-03-26T21:34:52.897024+0000","last_undegraded":"2024-03-26T21:34:52.897024+0000","last_fullsized":"2024-03-26T21:34:52.897024+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:29:58.537146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:29:58.537146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:34:53.970063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":324,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.638103+0000","last_change":"2024-03-26T21:34:19.761741+0000","last_active":"2024-03-26T21:34:52.638103+0000","last_peered":"2024-03-26T21:34:52.638103+0000","last_clean":"2024-03-26T21:34:52.638103+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:34:52.638103+0000","last_undegraded":"2024-03-26T21:34:52.638103+0000","last_fullsized":"2024-03-26T21:34:52.638103+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:34:19.761382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:19.761382+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.563424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.895758+0000","last_change":"2024-03-26T21:34:07.273064+0000","last_active":"2024-03-26T21:34:52.895758+0000","last_peered":"2024-03-26T21:34:52.895758+0000","last_clean":"2024-03-26T21:34:52.895758+0000","last_became_active":"2024-03-26T21:32:55.473354+0000","last_became_peered":"2024-03-26T21:32:55.473354+0000","last_unstale":"2024-03-26T21:34:52.895758+0000","last_undegraded":"2024-03-26T21:34:52.895758+0000","last_fullsized":"2024-03-26T21:34:52.895758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.272948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.272948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:31.339263+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":171,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-26T21:34:52.896787+0000","last_change":"2024-03-26T21:34:14.786736+0000","last_active":"2024-03-26T21:34:52.896787+0000","last_peered":"2024-03-26T21:34:52.896787+0000","last_clean":"2024-03-26T21:34:52.896787+0000","last_became_active":"2024-03-26T21:32:55.476367+0000","last_became_peered":"2024-03-26T21:32:55.476367+0000","last_unstale":"2024-03-26T21:34:52.896787+0000","last_undegraded":"2024-03-26T21:34:52.896787+0000","last_fullsized":"2024-03-26T21:34:52.896787+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:34:14.786394+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.786394+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.339829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":375,"num_read_kb":370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070545,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122780,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622372,"statfs":{"total":95995035648,"available":95869308928,"internally_reserved":0,"allocated":1392640,"data_stored":997932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364887,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122748,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622404,"statfs":{"total":95995035648,"available":95869341696,"internally_reserved":0,"allocated":1359872,"data_stored":995520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148588,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596564,"statfs":{"total":95995035648,"available":95842881536,"internally_reserved":0,"allocated":819200,"data_stored":407015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172764,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93572388,"statfs":{"total":95995035648,"available":95818125312,"internally_reserved":0,"allocated":1392640,"data_stored":997238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144372,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600780,"statfs":{"total":95995035648,"available":95847198720,"internally_reserved":0,"allocated":761856,"data_stored":405267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184672,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93560480,"statfs":{"total":95995035648,"available":95805931520,"internally_reserved":0,"allocated":806912,"data_stored":406056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":188283335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650393,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159260,"kb_used_data":784,"kb_used_omap":4,"kb_used_meta":158459,"kb_avail":93585892,"statfs":{"total":95995035648,"available":95831953408,"internally_reserved":0,"allocated":802816,"data_stored":407104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":162262691},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659226,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94412,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93650740,"statfs":{"total":95995035648,"available":95898357760,"internally_reserved":0,"allocated":786432,"data_stored":405836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:34:59.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph status --format=json 2024-03-26T21:34:59.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:59.374 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:59.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:34:59.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:34:59.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:34:59.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.633 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:59.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:34:59.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:34:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:34:59.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-26T21:34:59.857 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[40587]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:59.857 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[40587]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:34:59.857 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[40587]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:34:59.858 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[29406]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:59.858 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[29406]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:34:59.858 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:34:59 smithi055 ceph-mon[29406]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:34:59.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:34:59.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:34:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:34:59.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:34:59.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:34:59.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:59 smithi067 ceph-mon[33346]: from='client.24938 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:34:59.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:59 smithi067 ceph-mon[33346]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T21:34:59.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:34:59 smithi067 ceph-mon[33346]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T21:35:00.014 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:00.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.144 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:00.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:00.210 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:00.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:00.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:00.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:00.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:00.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.524 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:00.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:00.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:00.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:00.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:00.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:00.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:00 smithi067 ceph-mon[33346]: pgmap v267: 148 pgs: 1 creating+peering, 15 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-03-26T21:35:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:00 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:01.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.038 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:01.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:01.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:00 smithi055 ceph-mon[29406]: pgmap v267: 148 pgs: 1 creating+peering, 15 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-03-26T21:35:01.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:00 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:00 smithi055 ceph-mon[40587]: pgmap v267: 148 pgs: 1 creating+peering, 15 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 757 B/s rd, 0 op/s 2024-03-26T21:35:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:00 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:35:01.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:01.165 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:01.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:01.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:01.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:01.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:01.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.553 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:01.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:01.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:01.680 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:01.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:01.962 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:01.962 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:02.010 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:35:02.010 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:35:02.021 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.047 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:02.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:02.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:02.249 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:02.249 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:02.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:02.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:02.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:02.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:02.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:02.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:02.655 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:02.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:02.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:02.932 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:02.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:03.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:03.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:03.163 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.188 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:03.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:02 smithi067 ceph-mon[33346]: pgmap v269: 148 pgs: 1 creating+peering, 10 unknown, 137 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:35:03.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.314 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:03.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:02 smithi055 ceph-mon[40587]: pgmap v269: 148 pgs: 1 creating+peering, 10 unknown, 137 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:35:03.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:02 smithi055 ceph-mon[29406]: pgmap v269: 148 pgs: 1 creating+peering, 10 unknown, 137 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:35:03.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:03.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:03.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:03.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:03.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:03.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:03.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:03.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:03.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:03.951 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:03.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:04.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:04.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:04.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:04.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:04.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:04.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:04.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:04.592 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:04.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:04.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:04.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.847 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:04.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:04.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:04.974 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:04.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:05.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:04 smithi055 ceph-mon[29406]: pgmap v270: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:35:05.061 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:04 smithi055 ceph-mon[40587]: pgmap v270: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:35:05.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:05.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:35:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.101 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:05.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:05.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:05.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:04 smithi067 ceph-mon[33346]: pgmap v270: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:35:05.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:05.351 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:05.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:05.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.609 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:05.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:05.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:05.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:05.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:05.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:05.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:05.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:06.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:06.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.257 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:06.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:06.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:06.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:06.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:06.772 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:06.873 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:06.875 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:06.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:06.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:06.951 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-26T21:35:07.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:07.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:06 smithi055 ceph-mon[29406]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 663 B/s rd, 0 op/s 2024-03-26T21:35:07.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:06 smithi055 ceph-mon[40587]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 663 B/s rd, 0 op/s 2024-03-26T21:35:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:07.133 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:07.137 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:35:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.160 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:07.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:07.160 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:35:07.181 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:35:07.199 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:35:07.220 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:35:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:06 smithi067 ceph-mon[33346]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 663 B/s rd, 0 op/s 2024-03-26T21:35:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:35:07.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:07.240 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:35:07.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:07.261 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:35:07.281 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:35:07.282 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:35:07.282 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:35:07.282 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:35:07.283 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:35:07.283 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:35:07.283 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:35:07.283 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:35:07.283 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:35:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:07.388 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.413 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:07.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:07.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:07.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:07.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:07.667 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:07.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:07.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:07.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:07.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:07.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:07.963 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:35:08.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:08.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[29406]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:35:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:35:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2221817183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[40587]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:35:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:35:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2221817183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:08.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:08.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1783551829' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:35:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:07 smithi067 ceph-mon[33346]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T21:35:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:35:08.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2221817183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:08.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:08.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:08.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:08.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:08.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:08.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:08.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:08.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:08.688 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:08.692 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:08.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.818 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:08.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:08.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:08.920 DEBUG:teuthology.orchestra.run.smithi055:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-26T21:35:08.922 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T21:35:08.922 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T21:35:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:08.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:08.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:hints = 1 2024-03-26T21:35:08.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T21:35:08.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object prefix: benchmark_data_smithi055_110126 2024-03-26T21:35:08.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:35:08.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 0 0 0 0 0 0 - 0 2024-03-26T21:35:09.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:09.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: pgmap v273: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: pgmap v273: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:35:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:08 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:35:09.085 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:09.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:09.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: pgmap v273: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/153505042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T21:35:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:35:09.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:08 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:35:09.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:09.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:09.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:09.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.638 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:09.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:09.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:09.781 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:09.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:09.922 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:09.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:09.975 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 1 16 3145 3129 195.551 195.562 0.0124362 0.00508811 2024-03-26T21:35:10.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:10.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:10.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:35:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:35:10.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:09 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:35:10.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:10.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:35:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:35:10.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:35:10.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:35:10.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:09 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:35:10.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:10.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:10.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:10.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:35:10.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:10.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:10.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:10.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:10.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:10.976 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 2 16 5965 5949 185.782 176.25 0.00207134 0.00536817 2024-03-26T21:35:11.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:11.096 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:11.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:11.211 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[29406]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:35:11.211 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[40587]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:35:11.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:10 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:35:11.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:10 smithi067 ceph-mon[33346]: pgmap v275: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:35:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:10 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:35:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:10 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:35:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:10 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:35:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:10 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:35:11.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:11.372 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:11.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:11.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:11.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:11.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:11.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:11.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:11.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:11.927 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:11.977 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 3 16 9023 9007 187.541 191.125 0.00452952 0.00532229 2024-03-26T21:35:12.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.066 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:12.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:12.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:35:12.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:12 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:35:12.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:12.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:12.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:12.345 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:12.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:12.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:12 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:35:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:12 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:35:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:12 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:35:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:12 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:35:12.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:12.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:12.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:12.759 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:12.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:12.896 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:12.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:12.977 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 4 15 11965 11950 186.636 183.938 0.003725 0.00534915 2024-03-26T21:35:12.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:13.042 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:13.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:13.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: pgmap v276: 148 pgs: 148 active+clean; 141 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: pgmap v276: 148 pgs: 148 active+clean; 141 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:35:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:35:13.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:35:13.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:35:13.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:13 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:35:13.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:13.326 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:13.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:13.459 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:13.466 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:35:13.466 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:35:13.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:13287 8.f 8:fd208c5e:::benchmark_data_smithi055_110126_object13286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:35:13.470156+0000", 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.0032830530000000002, 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003321573, 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 13287 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:13.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470156+0000", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470156+0000", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470158+0000", 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.595e-06 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470180+0000", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1840000000000001e-05 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470194+0000", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4034e-05 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470202+0000", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5140000000000001e-06 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:13.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.470900+0000", 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00069757800000000002 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.471025+0000", 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012554100000000001 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.471307+0000", 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00028176900000000002 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:35:13.472552+0000", 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0012450880000000001 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 1 2024-03-26T21:35:13.471 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: pgmap v276: 148 pgs: 148 active+clean; 141 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:35:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:13 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:35:13.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:13.608 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:13.710 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:13.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:13.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:13.887 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:13.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:13.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:35:13.977 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 5 16 14584 14568 182.03 163.625 0.00671427 0.00548489 2024-03-26T21:35:13.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.021 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:14.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:14.122 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.171 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:14.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:14.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:35:14.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:35:14.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:14 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:35:14.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:14.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:14.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:35:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:14.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:35:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:35:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:35:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:35:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:35:14.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:35:14.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:14 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:35:14.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:14.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:14.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.758 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:14.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:14.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:14.905 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:14.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:14.958 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:35:14.958 INFO:teuthology.orchestra.run.smithi055.stdout:{"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":627,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":177,"num_osds":8,"num_up_osds":8,"osd_up_since":1711488774,"num_in_osds":8,"osd_in_since":1711488482,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":148}],"num_pgs":148,"num_pools":7,"num_objects":7734,"data_bytes":494475688,"bytes_used":2259390464,"bytes_avail":765700894720,"bytes_total":767960285184,"read_bytes_sec":1279,"write_bytes_sec":61710028,"read_op_per_sec":1,"write_op_per_sec":941},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.55:8443/","prometheus":"http://172.21.15.55:9283/"}},"servicemap":{"epoch":15,"modified":"2024-03-26T21:33:55.377574+0000","services":{"rgw":{"daemons":{"summary":"","14454":{"start_epoch":12,"start_stamp":"2024-03-26T21:28:41.991230+0000","gid":14454,"addr":"172.21.15.55:0/1460561447","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-2484-g42fc52b8 (42fc52b86b14d4f88c40d75eb7102e0ab3918f55) squid (dev)","ceph_version_short":"19.0.0-2484-g42fc52b8","container_hostname":"smithi055","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi055","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024","kernel_version":"5.14.0-432.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32494632","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"eb62fa79-4a7b-4ae9-96fd-a5bbf94f2522","zone_name":"default","zonegroup_id":"38ece41e-66c6-4da1-a4ba-c1eb119d3e56","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-26T21:35:14.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 6 16 16993 16977 176.766 150.562 0.00546512 0.00565185 2024-03-26T21:35:15.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:15.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:15.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.192 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:15.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:15.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[40587]: pgmap v277: 148 pgs: 148 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 942 op/s 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2833978564' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[29406]: pgmap v277: 148 pgs: 148 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 942 op/s 2024-03-26T21:35:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:15 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2833978564' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T21:35:15.327 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:15.429 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:15.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:15 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:35:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:15 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:35:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:15 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:35:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:15 smithi067 ceph-mon[33346]: pgmap v277: 148 pgs: 148 active+clean; 472 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 942 op/s 2024-03-26T21:35:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:15 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2833978564' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T21:35:15.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:15.629 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:15.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:15.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:15.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:15.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:15.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 7 16 19553 19537 174.37 160 0.00392779 0.00572977 2024-03-26T21:35:16.010 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-26T21:35:16.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:35:16.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.102 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:16.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:16.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 3.8 scrub starts 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 3.8 scrub ok 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:35:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 3.8 scrub starts 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 3.8 scrub ok 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:35:16.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:16.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:16.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:16.368 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:16.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 3.8 scrub starts 2024-03-26T21:35:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 3.8 scrub ok 2024-03-26T21:35:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:35:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:35:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:35:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:35:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:35:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:35:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:16.502 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:16.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.633 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:16.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:16.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:16.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:16.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:16.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:16.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:16.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 8 16 21571 21555 168.341 126.125 0.0270419 0.00592644 2024-03-26T21:35:17.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:17.025 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.059 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:17.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:17.106 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: pgmap v278: 148 pgs: 148 active+clean; 523 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 606 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3927707371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: pgmap v278: 148 pgs: 148 active+clean; 523 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 606 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:35:17.107 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:35:17.108 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:35:17.108 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:17 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3927707371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:17.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:17.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:17.214 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:35:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:35:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:35:17.215 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:35:17.239 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:35:17.241 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:17.242 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:17.281 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:35:17.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:17.323 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:17.327 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:35:17.365 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:35:17.369 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:35:17.370 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:35:17.397 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:17.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.473 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:17.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: pgmap v278: 148 pgs: 148 active+clean; 523 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 606 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-26T21:35:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:35:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:35:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:35:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:35:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:35:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:17 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3927707371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:17.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:17.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:17.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.756 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:17.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:17.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:17.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:17.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 9 16 23830 23814 165.323 141.188 0.00683387 0.00604239 2024-03-26T21:35:17.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:18.030 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:18.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:18.169 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:18.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:18.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:18.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:35:18.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:18.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:35:18.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:35:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:35:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:35:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:35:18.405 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:18.445 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:18 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:35:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:18 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:35:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:18 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:35:18.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:18.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:18.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:18.721 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:18.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:18.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:18.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:18.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:18.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 10 16 26212 26196 163.675 148.875 0.00636512 0.00610448 2024-03-26T21:35:19.006 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:19.009 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:19.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:19.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:19.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:19.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[29406]: pgmap v279: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[40587]: pgmap v279: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:35:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:19 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:35:19.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:19.417 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:19 smithi067 ceph-mon[33346]: pgmap v279: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-26T21:35:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:19 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:35:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:19 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:35:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:19 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:35:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:19 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:35:19.519 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:19.559 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:19.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:19.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:19.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:19.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:19.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:19.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:19.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:19.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 11 16 28485 28469 161.709 142.062 0.0084011 0.00617761 2024-03-26T21:35:20.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:20.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:20.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:20.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:20.376 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:20.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:20.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:35:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:20.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:20.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.651 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:20.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:20.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:20.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:20.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:20.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:20.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 12 16 30442 30426 158.426 122.312 0.00471711 0.00630613 2024-03-26T21:35:21.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:21.069 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:21.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:21.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:21.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:35:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:35:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: pgmap v280: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 895 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: pgmap v280: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 895 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:35:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:35:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:35:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:35:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:35:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:21 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:35:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.340 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:21.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: pgmap v280: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 895 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:35:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:35:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:35:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:35:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:21 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:35:21.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:21.713 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:21.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:21.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:21.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:21.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:21.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 13 15 32224 32209 154.811 111.438 0.00482408 0.00645378 2024-03-26T21:35:21.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:22.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:22.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:22.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:22.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:22.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:22.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:22.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:22 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:35:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:22 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:35:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:22 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:22 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:22 smithi067 ceph-mon[33346]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:35:22.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:22.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:35:22.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:35:22.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:22.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:22.556 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[29406]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:35:22.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:35:22.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:35:22.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:22.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:22.557 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:22 smithi055 ceph-mon[40587]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T21:35:22.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:22.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:22.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:22.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:22.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:22.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:22.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:22.978 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 14 16 33770 33754 150.651 96.5625 0.00870463 0.00663198 2024-03-26T21:35:23.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.096 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:23.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:23.197 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:23.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:23.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:23.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:23.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:23 smithi067 ceph-mon[33346]: pgmap v282: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T21:35:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:23 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:35:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:23 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:35:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:23.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[40587]: pgmap v282: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T21:35:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:35:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:35:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[29406]: pgmap v282: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T21:35:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:35:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:23 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:35:23.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:23.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:23.726 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:23.749 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:23.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:23.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:23.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:23.979 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 15 16 35256 35240 146.799 92.875 0.00820299 0.0068067 2024-03-26T21:35:23.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:24.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:24.170 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:24.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:24.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:24.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:24.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:24 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:35:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:24 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:35:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:24 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:35:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:24 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:35:24.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:35:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:35:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:35:24.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:35:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:35:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:35:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:35:24.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:24 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:35:24.572 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:24.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:24.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:24.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:24.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:24.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:24.979 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 16 16 36847 36831 143.839 99.4375 0.0021246 0.00694245 2024-03-26T21:35:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:24.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:25.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:25.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:25.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.263 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:25.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:25.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:25.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[29406]: pgmap v283: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[40587]: pgmap v283: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:35:25.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:35:25.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:35:25.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:25 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:35:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:25 smithi067 ceph-mon[33346]: pgmap v283: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-26T21:35:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:25 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:35:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:25 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:35:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:25 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:35:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:25 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:35:25.498 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:25.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:25.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:25.659 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:25.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:25.784 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:25.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:25.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:25.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:25.979 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 17 16 38389 38373 141.047 96.375 0.00559315 0.00708259 2024-03-26T21:35:26.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:26.050 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:26.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:26.193 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:26.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:26.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:26.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:26.471 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:26 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:35:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:26 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:35:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:26 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:35:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:26 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:35:26.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:35:26.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:35:26.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:26 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:35:26.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:26.610 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:26.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:26.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:26.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:26.880 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:26.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:26.979 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 18 16 39898 39882 138.45 94.3125 0.00488676 0.00721725 2024-03-26T21:35:26.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:27.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:27.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:27.148 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:27.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:27.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:27 smithi067 ceph-mon[33346]: pgmap v284: 148 pgs: 148 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-26T21:35:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:27 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:35:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:27 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:35:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:27 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:35:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:27 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:35:27.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[40587]: pgmap v284: 148 pgs: 148 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-26T21:35:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[29406]: pgmap v284: 148 pgs: 148 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:35:27.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:27 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:35:27.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:27.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:27.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:27.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:27.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:27.979 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 19 16 41833 41817 137.528 120.938 0.0080688 0.00726568 2024-03-26T21:35:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:27.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:27.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:28.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:28.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:28.273 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:28.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:35:28.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:28.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:35:28.388 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:35:28.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:35:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:35:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:28 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:35:28.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:28.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:28.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:28.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:28.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:28.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:28.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:28.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:35:28.988265+0000 min lat: 0.0012867 max lat: 0.203486 avg lat: 0.007315 2024-03-26T21:35:28.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:35:28.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 20 16 43741 43725 136.608 119.25 0.00640445 0.007315 2024-03-26T21:35:29.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:29.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:29.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:29.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:29.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:29.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:29.369 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:35:29.369 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: pgmap v285: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T21:35:29.369 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: pgmap v285: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T21:35:29.370 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:35:29.371 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:29 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:35:29.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:35:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: pgmap v285: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-26T21:35:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:35:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:35:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T21:35:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:35:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:35:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:35:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:29 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:35:29.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:29.549 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:29.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:29.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:29.787 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:35:29.787 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:35:29.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.823 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:29.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:29.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:29.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:29.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:29.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 21 16 45199 45183 134.442 91.125 0.0125517 0.00743163 2024-03-26T21:35:30.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:30.098 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:35:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:30.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:30.108 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:35:30.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:30.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:30.368 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:30.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:30.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:30 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:35:30.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:30 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:35:30.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:30 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:35:30.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:30 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:35:30.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:30.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:35:30.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:35:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:30 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:35:30.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:30.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:30.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:35:30.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:30.809 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:30.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:30.958 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:30.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:30.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 22 16 46701 46685 132.598 93.875 0.00507526 0.00753491 2024-03-26T21:35:31.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:31.087 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:31.138 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[29406]: pgmap v287: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-26T21:35:31.138 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[29406]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:31.138 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:35:31.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.223 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:31.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:31.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:31.456 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[40587]: pgmap v287: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-26T21:35:31.456 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[40587]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:31.457 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:35:31.457 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:35:31.457 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:31 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:35:31.457 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:31 smithi067 ceph-mon[33346]: pgmap v287: 148 pgs: 148 active+clean; 2.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-26T21:35:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:31 smithi067 ceph-mon[33346]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:31 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:35:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:31 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:35:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.502 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:31.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:31.625 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:31.670 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-26T21:35:29.408773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2599885224,"num_objects":39860,"num_object_clones":0,"num_object_copies":119580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39860,"num_whiteouts":0,"num_read":1331,"num_read_kb":1177,"num_write":40277,"num_write_kb":2540829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40329,"ondisk_log_size":40329,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8712716,"kb_used_data":7459976,"kb_used_omap":15,"kb_used_meta":1252656,"kb_avail":741248500,"statfs":{"total":767960285184,"available":759038464000,"internally_reserved":0,"allocated":7639015424,"data_stored":7635658189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1282720526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1211891712,"num_objects":18492,"num_object_clones":0,"num_object_copies":55476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18492,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18492,"num_write_kb":1183488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.336323"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":164,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127478+0000","last_change":"2024-03-26T21:35:18.215494+0000","last_active":"2024-03-26T21:35:28.127478+0000","last_peered":"2024-03-26T21:35:28.127478+0000","last_clean":"2024-03-26T21:35:28.127478+0000","last_became_active":"2024-03-26T21:32:55.466152+0000","last_became_peered":"2024-03-26T21:32:55.466152+0000","last_unstale":"2024-03-26T21:35:28.127478+0000","last_undegraded":"2024-03-26T21:35:28.127478+0000","last_fullsized":"2024-03-26T21:35:28.127478+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:18.215367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:18.215367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:47.706122+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144582+0000","last_change":"2024-03-26T21:34:20.509053+0000","last_active":"2024-03-26T21:35:21.144582+0000","last_peered":"2024-03-26T21:35:21.144582+0000","last_clean":"2024-03-26T21:35:21.144582+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:35:21.144582+0000","last_undegraded":"2024-03-26T21:35:21.144582+0000","last_fullsized":"2024-03-26T21:35:21.144582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.508804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.508804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:37.144539+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121536+0000","last_change":"2024-03-26T21:32:14.926305+0000","last_active":"2024-03-26T21:35:21.121536+0000","last_peered":"2024-03-26T21:35:21.121536+0000","last_clean":"2024-03-26T21:35:21.121536+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:35:21.121536+0000","last_undegraded":"2024-03-26T21:35:21.121536+0000","last_fullsized":"2024-03-26T21:35:21.121536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:14.926185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:32:14.926185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":184,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127014+0000","last_change":"2024-03-26T21:34:20.720386+0000","last_active":"2024-03-26T21:35:28.127014+0000","last_peered":"2024-03-26T21:35:28.127014+0000","last_clean":"2024-03-26T21:35:28.127014+0000","last_became_active":"2024-03-26T21:32:55.468198+0000","last_became_peered":"2024-03-26T21:32:55.468198+0000","last_unstale":"2024-03-26T21:35:28.127014+0000","last_undegraded":"2024-03-26T21:35:28.127014+0000","last_fullsized":"2024-03-26T21:35:28.127014+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:20.720301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.720301+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.145805+0000","last_change":"2024-03-26T21:34:24.368408+0000","last_active":"2024-03-26T21:35:21.145805+0000","last_peered":"2024-03-26T21:35:21.145805+0000","last_clean":"2024-03-26T21:35:21.145805+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:35:21.145805+0000","last_undegraded":"2024-03-26T21:35:21.145805+0000","last_fullsized":"2024-03-26T21:35:21.145805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.368161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.690462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":164,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.126820+0000","last_change":"2024-03-26T21:34:27.847069+0000","last_active":"2024-03-26T21:35:28.126820+0000","last_peered":"2024-03-26T21:35:28.126820+0000","last_clean":"2024-03-26T21:35:28.126820+0000","last_became_active":"2024-03-26T21:32:55.468736+0000","last_became_peered":"2024-03-26T21:32:55.468736+0000","last_unstale":"2024-03-26T21:35:28.126820+0000","last_undegraded":"2024-03-26T21:35:28.126820+0000","last_fullsized":"2024-03-26T21:35:28.126820+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.846942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.846942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.712678+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142194+0000","last_change":"2024-03-26T21:34:30.380605+0000","last_active":"2024-03-26T21:35:21.142194+0000","last_peered":"2024-03-26T21:35:21.142194+0000","last_clean":"2024-03-26T21:35:21.142194+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:35:21.142194+0000","last_undegraded":"2024-03-26T21:35:21.142194+0000","last_fullsized":"2024-03-26T21:35:21.142194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:30.379661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.379661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.318868+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":280,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:24.162274+0000","last_change":"2024-03-26T21:35:24.162334+0000","last_active":"2024-03-26T21:35:24.162274+0000","last_peered":"2024-03-26T21:35:24.162274+0000","last_clean":"2024-03-26T21:35:24.162274+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:35:24.162274+0000","last_undegraded":"2024-03-26T21:35:24.162274+0000","last_fullsized":"2024-03-26T21:35:24.162274+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:35:24.162191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:24.162191+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.771496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:25.946205+0000","last_change":"2024-03-26T21:35:25.946247+0000","last_active":"2024-03-26T21:35:25.946205+0000","last_peered":"2024-03-26T21:35:25.946205+0000","last_clean":"2024-03-26T21:35:25.946205+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:35:25.946205+0000","last_undegraded":"2024-03-26T21:35:25.946205+0000","last_fullsized":"2024-03-26T21:35:25.946205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.946125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.946125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.103970+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.076506+0000","last_change":"2024-03-26T21:34:31.792345+0000","last_active":"2024-03-26T21:35:21.076506+0000","last_peered":"2024-03-26T21:35:21.076506+0000","last_clean":"2024-03-26T21:35:21.076506+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:35:21.076506+0000","last_undegraded":"2024-03-26T21:35:21.076506+0000","last_fullsized":"2024-03-26T21:35:21.076506+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:31.792268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.792268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.823508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077870+0000","last_change":"2024-03-26T21:33:22.738629+0000","last_active":"2024-03-26T21:35:21.077870+0000","last_peered":"2024-03-26T21:35:21.077870+0000","last_clean":"2024-03-26T21:35:21.077870+0000","last_became_active":"2024-03-26T21:32:55.465246+0000","last_became_peered":"2024-03-26T21:32:55.465246+0000","last_unstale":"2024-03-26T21:35:21.077870+0000","last_undegraded":"2024-03-26T21:35:21.077870+0000","last_fullsized":"2024-03-26T21:35:21.077870+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.738510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.738510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"167'13","reported_seq":188,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127435+0000","last_change":"2024-03-26T21:34:30.923713+0000","last_active":"2024-03-26T21:35:28.127435+0000","last_peered":"2024-03-26T21:35:28.127435+0000","last_clean":"2024-03-26T21:35:28.127435+0000","last_became_active":"2024-03-26T21:32:55.480936+0000","last_became_peered":"2024-03-26T21:32:55.480936+0000","last_unstale":"2024-03-26T21:35:28.127435+0000","last_undegraded":"2024-03-26T21:35:28.127435+0000","last_fullsized":"2024-03-26T21:35:28.127435+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.923562+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:40.154880+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078244+0000","last_change":"2024-03-26T21:34:22.762001+0000","last_active":"2024-03-26T21:35:21.078244+0000","last_peered":"2024-03-26T21:35:21.078244+0000","last_clean":"2024-03-26T21:35:21.078244+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:35:21.078244+0000","last_undegraded":"2024-03-26T21:35:21.078244+0000","last_fullsized":"2024-03-26T21:35:21.078244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:22.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:34:22.761871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.252793+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":140,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078872+0000","last_change":"2024-03-26T21:34:23.724557+0000","last_active":"2024-03-26T21:35:21.078872+0000","last_peered":"2024-03-26T21:35:21.078872+0000","last_clean":"2024-03-26T21:35:21.078872+0000","last_became_active":"2024-03-26T21:32:20.904068+0000","last_became_peered":"2024-03-26T21:32:20.904068+0000","last_unstale":"2024-03-26T21:35:21.078872+0000","last_undegraded":"2024-03-26T21:35:21.078872+0000","last_fullsized":"2024-03-26T21:35:21.078872+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.724419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.724419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.040112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077048+0000","last_change":"2024-03-26T21:33:25.751247+0000","last_active":"2024-03-26T21:35:21.077048+0000","last_peered":"2024-03-26T21:35:21.077048+0000","last_clean":"2024-03-26T21:35:21.077048+0000","last_became_active":"2024-03-26T21:32:55.466491+0000","last_became_peered":"2024-03-26T21:32:55.466491+0000","last_unstale":"2024-03-26T21:35:21.077048+0000","last_undegraded":"2024-03-26T21:35:21.077048+0000","last_fullsized":"2024-03-26T21:35:21.077048+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:25.751149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:25.751149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":143,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127455+0000","last_change":"2024-03-26T21:34:11.780408+0000","last_active":"2024-03-26T21:35:28.127455+0000","last_peered":"2024-03-26T21:35:28.127455+0000","last_clean":"2024-03-26T21:35:28.127455+0000","last_became_active":"2024-03-26T21:32:55.476272+0000","last_became_peered":"2024-03-26T21:32:55.476272+0000","last_unstale":"2024-03-26T21:35:28.127455+0000","last_undegraded":"2024-03-26T21:35:28.127455+0000","last_fullsized":"2024-03-26T21:35:28.127455+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:34:11.780127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.780127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":121,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.129745+0000","last_change":"2024-03-26T21:34:06.784378+0000","last_active":"2024-03-26T21:35:28.129745+0000","last_peered":"2024-03-26T21:35:28.129745+0000","last_clean":"2024-03-26T21:35:28.129745+0000","last_became_active":"2024-03-26T21:32:55.467335+0000","last_became_peered":"2024-03-26T21:32:55.467335+0000","last_unstale":"2024-03-26T21:35:28.129745+0000","last_undegraded":"2024-03-26T21:35:28.129745+0000","last_fullsized":"2024-03-26T21:35:28.129745+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:06.784228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:06.784228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144992+0000","last_change":"2024-03-26T21:35:16.465328+0000","last_active":"2024-03-26T21:35:21.144992+0000","last_peered":"2024-03-26T21:35:21.144992+0000","last_clean":"2024-03-26T21:35:21.144992+0000","last_became_active":"2024-03-26T21:32:55.472908+0000","last_became_peered":"2024-03-26T21:32:55.472908+0000","last_unstale":"2024-03-26T21:35:21.144992+0000","last_undegraded":"2024-03-26T21:35:21.144992+0000","last_fullsized":"2024-03-26T21:35:21.144992+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:16.465203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:16.465203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:22.841819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121302+0000","last_change":"2024-03-26T21:34:27.088958+0000","last_active":"2024-03-26T21:35:21.121302+0000","last_peered":"2024-03-26T21:35:21.121302+0000","last_clean":"2024-03-26T21:35:21.121302+0000","last_became_active":"2024-03-26T21:32:55.479242+0000","last_became_peered":"2024-03-26T21:32:55.479242+0000","last_unstale":"2024-03-26T21:35:21.121302+0000","last_undegraded":"2024-03-26T21:35:21.121302+0000","last_fullsized":"2024-03-26T21:35:21.121302+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.088840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.088840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:49.279253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":288,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082962+0000","last_change":"2024-03-26T21:35:13.081542+0000","last_active":"2024-03-26T21:35:21.082962+0000","last_peered":"2024-03-26T21:35:21.082962+0000","last_clean":"2024-03-26T21:35:21.082962+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:35:21.082962+0000","last_undegraded":"2024-03-26T21:35:21.082962+0000","last_fullsized":"2024-03-26T21:35:21.082962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:35:13.081138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.081138+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:22.799380+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":240,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142491+0000","last_change":"2024-03-26T21:32:30.360160+0000","last_active":"2024-03-26T21:35:21.142491+0000","last_peered":"2024-03-26T21:35:21.142491+0000","last_clean":"2024-03-26T21:35:21.142491+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:35:21.142491+0000","last_undegraded":"2024-03-26T21:35:21.142491+0000","last_fullsized":"2024-03-26T21:35:21.142491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:30.360050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:30.360050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":164,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127910+0000","last_change":"2024-03-26T21:34:25.862079+0000","last_active":"2024-03-26T21:35:28.127910+0000","last_peered":"2024-03-26T21:35:28.127910+0000","last_clean":"2024-03-26T21:35:28.127910+0000","last_became_active":"2024-03-26T21:32:55.465947+0000","last_became_peered":"2024-03-26T21:32:55.465947+0000","last_unstale":"2024-03-26T21:35:28.127910+0000","last_undegraded":"2024-03-26T21:35:28.127910+0000","last_fullsized":"2024-03-26T21:35:28.127910+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.861995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.861995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.431656+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":176,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127782+0000","last_change":"2024-03-26T21:35:25.096014+0000","last_active":"2024-03-26T21:35:28.127782+0000","last_peered":"2024-03-26T21:35:28.127782+0000","last_clean":"2024-03-26T21:35:28.127782+0000","last_became_active":"2024-03-26T21:32:55.477250+0000","last_became_peered":"2024-03-26T21:32:55.477250+0000","last_unstale":"2024-03-26T21:35:28.127782+0000","last_undegraded":"2024-03-26T21:35:28.127782+0000","last_fullsized":"2024-03-26T21:35:28.127782+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.095921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.095921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.118555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142844+0000","last_change":"2024-03-26T21:34:09.286797+0000","last_active":"2024-03-26T21:35:21.142844+0000","last_peered":"2024-03-26T21:35:21.142844+0000","last_clean":"2024-03-26T21:35:21.142844+0000","last_became_active":"2024-03-26T21:32:55.470311+0000","last_became_peered":"2024-03-26T21:32:55.470311+0000","last_unstale":"2024-03-26T21:35:21.142844+0000","last_undegraded":"2024-03-26T21:35:21.142844+0000","last_fullsized":"2024-03-26T21:35:21.142844+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:09.286636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.286636+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.511715+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":256,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082485+0000","last_change":"2024-03-26T21:34:25.733995+0000","last_active":"2024-03-26T21:35:21.082485+0000","last_peered":"2024-03-26T21:35:21.082485+0000","last_clean":"2024-03-26T21:35:21.082485+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:35:21.082485+0000","last_undegraded":"2024-03-26T21:35:21.082485+0000","last_fullsized":"2024-03-26T21:35:21.082485+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:34:25.733888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.733888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:27.393061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"134'10","reported_seq":357,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077765+0000","last_change":"2024-03-26T21:34:27.049262+0000","last_active":"2024-03-26T21:35:21.077765+0000","last_peered":"2024-03-26T21:35:21.077765+0000","last_clean":"2024-03-26T21:35:21.077765+0000","last_became_active":"2024-03-26T21:32:55.477677+0000","last_became_peered":"2024-03-26T21:32:55.477677+0000","last_unstale":"2024-03-26T21:35:21.077765+0000","last_undegraded":"2024-03-26T21:35:21.077765+0000","last_fullsized":"2024-03-26T21:35:21.077765+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:27.048989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.048989+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.859023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":270,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144675+0000","last_change":"2024-03-26T21:35:12.512389+0000","last_active":"2024-03-26T21:35:21.144675+0000","last_peered":"2024-03-26T21:35:21.144675+0000","last_clean":"2024-03-26T21:35:21.144675+0000","last_became_active":"2024-03-26T21:32:55.469277+0000","last_became_peered":"2024-03-26T21:32:55.469277+0000","last_unstale":"2024-03-26T21:35:21.144675+0000","last_undegraded":"2024-03-26T21:35:21.144675+0000","last_fullsized":"2024-03-26T21:35:21.144675+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:35:12.511752+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.511752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:16.863479+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":263,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121908+0000","last_change":"2024-03-26T21:32:22.112186+0000","last_active":"2024-03-26T21:35:21.121908+0000","last_peered":"2024-03-26T21:35:21.121908+0000","last_clean":"2024-03-26T21:35:21.121908+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:35:21.121908+0000","last_undegraded":"2024-03-26T21:35:21.121908+0000","last_fullsized":"2024-03-26T21:35:21.121908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:32:22.112001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.112001+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":256,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144866+0000","last_change":"2024-03-26T21:35:08.399748+0000","last_active":"2024-03-26T21:35:21.144866+0000","last_peered":"2024-03-26T21:35:21.144866+0000","last_clean":"2024-03-26T21:35:21.144866+0000","last_became_active":"2024-03-26T21:32:55.465456+0000","last_became_peered":"2024-03-26T21:32:55.465456+0000","last_unstale":"2024-03-26T21:35:21.144866+0000","last_undegraded":"2024-03-26T21:35:21.144866+0000","last_fullsized":"2024-03-26T21:35:21.144866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:08.399643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.399643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:25.430769+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.122052+0000","last_change":"2024-03-26T21:35:10.003783+0000","last_active":"2024-03-26T21:35:21.122052+0000","last_peered":"2024-03-26T21:35:21.122052+0000","last_clean":"2024-03-26T21:35:21.122052+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:35:21.122052+0000","last_undegraded":"2024-03-26T21:35:21.122052+0000","last_fullsized":"2024-03-26T21:35:21.122052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:10.003660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:35:10.003660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:27.965246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143454+0000","last_change":"2024-03-26T21:35:20.257839+0000","last_active":"2024-03-26T21:35:21.143454+0000","last_peered":"2024-03-26T21:35:21.143454+0000","last_clean":"2024-03-26T21:35:21.143454+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:35:21.143454+0000","last_undegraded":"2024-03-26T21:35:21.143454+0000","last_fullsized":"2024-03-26T21:35:21.143454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:20.257696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:20.257696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:43.449764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"167'16","reported_seq":288,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077672+0000","last_change":"2024-03-26T21:34:28.014694+0000","last_active":"2024-03-26T21:35:21.077672+0000","last_peered":"2024-03-26T21:35:21.077672+0000","last_clean":"2024-03-26T21:35:21.077672+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:35:21.077672+0000","last_undegraded":"2024-03-26T21:35:21.077672+0000","last_fullsized":"2024-03-26T21:35:21.077672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'16","last_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.014533+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.217243+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080445+0000","last_change":"2024-03-26T21:33:22.621399+0000","last_active":"2024-03-26T21:35:21.080445+0000","last_peered":"2024-03-26T21:35:21.080445+0000","last_clean":"2024-03-26T21:35:21.080445+0000","last_became_active":"2024-03-26T21:32:55.471141+0000","last_became_peered":"2024-03-26T21:32:55.471141+0000","last_unstale":"2024-03-26T21:35:21.080445+0000","last_undegraded":"2024-03-26T21:35:21.080445+0000","last_fullsized":"2024-03-26T21:35:21.080445+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.621257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.621257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077539+0000","last_change":"2024-03-26T21:34:23.051204+0000","last_active":"2024-03-26T21:35:21.077539+0000","last_peered":"2024-03-26T21:35:21.077539+0000","last_clean":"2024-03-26T21:35:21.077539+0000","last_became_active":"2024-03-26T21:32:55.469690+0000","last_became_peered":"2024-03-26T21:32:55.469690+0000","last_unstale":"2024-03-26T21:35:21.077539+0000","last_undegraded":"2024-03-26T21:35:21.077539+0000","last_fullsized":"2024-03-26T21:35:21.077539+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.051104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.051104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.773972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121689+0000","last_change":"2024-03-26T21:32:36.887352+0000","last_active":"2024-03-26T21:35:21.121689+0000","last_peered":"2024-03-26T21:35:21.121689+0000","last_clean":"2024-03-26T21:35:21.121689+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:35:21.121689+0000","last_undegraded":"2024-03-26T21:35:21.121689+0000","last_fullsized":"2024-03-26T21:35:21.121689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:36.887160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.887160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143543+0000","last_change":"2024-03-26T21:34:25.343996+0000","last_active":"2024-03-26T21:35:21.143543+0000","last_peered":"2024-03-26T21:35:21.143543+0000","last_clean":"2024-03-26T21:35:21.143543+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:35:21.143543+0000","last_undegraded":"2024-03-26T21:35:21.143543+0000","last_fullsized":"2024-03-26T21:35:21.143543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:34:25.343633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.343633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.287164+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":239,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.146407+0000","last_change":"2024-03-26T21:32:29.393768+0000","last_active":"2024-03-26T21:35:21.146407+0000","last_peered":"2024-03-26T21:35:21.146407+0000","last_clean":"2024-03-26T21:35:21.146407+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:35:21.146407+0000","last_undegraded":"2024-03-26T21:35:21.146407+0000","last_fullsized":"2024-03-26T21:35:21.146407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:29.393601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.393601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081421+0000","last_change":"2024-03-26T21:35:14.435344+0000","last_active":"2024-03-26T21:35:21.081421+0000","last_peered":"2024-03-26T21:35:21.081421+0000","last_clean":"2024-03-26T21:35:21.081421+0000","last_became_active":"2024-03-26T21:32:55.474210+0000","last_became_peered":"2024-03-26T21:32:55.474210+0000","last_unstale":"2024-03-26T21:35:21.081421+0000","last_undegraded":"2024-03-26T21:35:21.081421+0000","last_fullsized":"2024-03-26T21:35:21.081421+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.435219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.435219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.974188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.076472+0000","last_change":"2024-03-26T21:33:30.764236+0000","last_active":"2024-03-26T21:35:21.076472+0000","last_peered":"2024-03-26T21:35:21.076472+0000","last_clean":"2024-03-26T21:35:21.076472+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:35:21.076472+0000","last_undegraded":"2024-03-26T21:35:21.076472+0000","last_fullsized":"2024-03-26T21:35:21.076472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.763924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.763924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":274,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:22.254412+0000","last_change":"2024-03-26T21:35:22.254470+0000","last_active":"2024-03-26T21:35:22.254412+0000","last_peered":"2024-03-26T21:35:22.254412+0000","last_clean":"2024-03-26T21:35:22.254412+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:35:22.254412+0000","last_undegraded":"2024-03-26T21:35:22.254412+0000","last_fullsized":"2024-03-26T21:35:22.254412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:35:22.254300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:22.254300+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:46.043692+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":239,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.072702+0000","last_change":"2024-03-26T21:32:26.906415+0000","last_active":"2024-03-26T21:35:21.072702+0000","last_peered":"2024-03-26T21:35:21.072702+0000","last_clean":"2024-03-26T21:35:21.072702+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:35:21.072702+0000","last_undegraded":"2024-03-26T21:35:21.072702+0000","last_fullsized":"2024-03-26T21:35:21.072702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.906307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.906307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.146207+0000","last_change":"2024-03-26T21:35:19.231094+0000","last_active":"2024-03-26T21:35:21.146207+0000","last_peered":"2024-03-26T21:35:21.146207+0000","last_clean":"2024-03-26T21:35:21.146207+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:35:21.146207+0000","last_undegraded":"2024-03-26T21:35:21.146207+0000","last_fullsized":"2024-03-26T21:35:21.146207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.230859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.230859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:42.696009+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081076+0000","last_change":"2024-03-26T21:33:17.583457+0000","last_active":"2024-03-26T21:35:21.081076+0000","last_peered":"2024-03-26T21:35:21.081076+0000","last_clean":"2024-03-26T21:35:21.081076+0000","last_became_active":"2024-03-26T21:32:55.470351+0000","last_became_peered":"2024-03-26T21:32:55.470351+0000","last_unstale":"2024-03-26T21:35:21.081076+0000","last_undegraded":"2024-03-26T21:35:21.081076+0000","last_fullsized":"2024-03-26T21:35:21.081076+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:17.583338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:17.583338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":275,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.145236+0000","last_change":"2024-03-26T21:34:13.533465+0000","last_active":"2024-03-26T21:35:21.145236+0000","last_peered":"2024-03-26T21:35:21.145236+0000","last_clean":"2024-03-26T21:35:21.145236+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:35:21.145236+0000","last_undegraded":"2024-03-26T21:35:21.145236+0000","last_fullsized":"2024-03-26T21:35:21.145236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:34:13.533314+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.533314+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.625607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.074741+0000","last_change":"2024-03-26T21:35:13.297009+0000","last_active":"2024-03-26T21:35:21.074741+0000","last_peered":"2024-03-26T21:35:21.074741+0000","last_clean":"2024-03-26T21:35:21.074741+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:35:21.074741+0000","last_undegraded":"2024-03-26T21:35:21.074741+0000","last_fullsized":"2024-03-26T21:35:21.074741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.296881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:28.501742+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144153+0000","last_change":"2024-03-26T21:34:25.554360+0000","last_active":"2024-03-26T21:35:21.144153+0000","last_peered":"2024-03-26T21:35:21.144153+0000","last_clean":"2024-03-26T21:35:21.144153+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:35:21.144153+0000","last_undegraded":"2024-03-26T21:35:21.144153+0000","last_fullsized":"2024-03-26T21:35:21.144153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.554230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.554230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.463566+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":179,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.149013+0000","last_change":"2024-03-26T21:35:28.149056+0000","last_active":"2024-03-26T21:35:28.149013+0000","last_peered":"2024-03-26T21:35:28.149013+0000","last_clean":"2024-03-26T21:35:28.149013+0000","last_became_active":"2024-03-26T21:32:55.467375+0000","last_became_peered":"2024-03-26T21:32:55.467375+0000","last_unstale":"2024-03-26T21:35:28.149013+0000","last_undegraded":"2024-03-26T21:35:28.149013+0000","last_fullsized":"2024-03-26T21:35:28.149013+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:28.148936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.148936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:55.503851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":152,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127707+0000","last_change":"2024-03-26T21:34:10.824961+0000","last_active":"2024-03-26T21:35:28.127707+0000","last_peered":"2024-03-26T21:35:28.127707+0000","last_clean":"2024-03-26T21:35:28.127707+0000","last_became_active":"2024-03-26T21:32:55.477977+0000","last_became_peered":"2024-03-26T21:32:55.477977+0000","last_unstale":"2024-03-26T21:35:28.127707+0000","last_undegraded":"2024-03-26T21:35:28.127707+0000","last_fullsized":"2024-03-26T21:35:28.127707+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:34:10.824829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.824829+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.688410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121566+0000","last_change":"2024-03-26T21:35:11.986871+0000","last_active":"2024-03-26T21:35:21.121566+0000","last_peered":"2024-03-26T21:35:21.121566+0000","last_clean":"2024-03-26T21:35:21.121566+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:35:21.121566+0000","last_undegraded":"2024-03-26T21:35:21.121566+0000","last_fullsized":"2024-03-26T21:35:21.121566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:35:11.986741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.986741+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:28.662140+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":343,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077997+0000","last_change":"2024-03-26T21:34:11.674348+0000","last_active":"2024-03-26T21:35:21.077997+0000","last_peered":"2024-03-26T21:35:21.077997+0000","last_clean":"2024-03-26T21:35:21.077997+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:35:21.077997+0000","last_undegraded":"2024-03-26T21:35:21.077997+0000","last_fullsized":"2024-03-26T21:35:21.077997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:34:11.674203+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.674203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:32.191741+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.079951+0000","last_change":"2024-03-26T21:35:19.470998+0000","last_active":"2024-03-26T21:35:21.079951+0000","last_peered":"2024-03-26T21:35:21.079951+0000","last_clean":"2024-03-26T21:35:21.079951+0000","last_became_active":"2024-03-26T21:32:55.473442+0000","last_became_peered":"2024-03-26T21:32:55.473442+0000","last_unstale":"2024-03-26T21:35:21.079951+0000","last_undegraded":"2024-03-26T21:35:21.079951+0000","last_fullsized":"2024-03-26T21:35:21.079951+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.470873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.470873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.799000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"166'15","reported_seq":275,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143259+0000","last_change":"2024-03-26T21:34:27.404044+0000","last_active":"2024-03-26T21:35:21.143259+0000","last_peered":"2024-03-26T21:35:21.143259+0000","last_clean":"2024-03-26T21:35:21.143259+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:35:21.143259+0000","last_undegraded":"2024-03-26T21:35:21.143259+0000","last_fullsized":"2024-03-26T21:35:21.143259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.403785+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.739092+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":159,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.129163+0000","last_change":"2024-03-26T21:32:55.466193+0000","last_active":"2024-03-26T21:35:28.129163+0000","last_peered":"2024-03-26T21:35:28.129163+0000","last_clean":"2024-03-26T21:35:28.129163+0000","last_became_active":"2024-03-26T21:32:55.465075+0000","last_became_peered":"2024-03-26T21:32:55.465075+0000","last_unstale":"2024-03-26T21:35:28.129163+0000","last_undegraded":"2024-03-26T21:35:28.129163+0000","last_fullsized":"2024-03-26T21:35:28.129163+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:30:25.185846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:30:25.185846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":337,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143525+0000","last_change":"2024-03-26T21:32:36.313231+0000","last_active":"2024-03-26T21:35:21.143525+0000","last_peered":"2024-03-26T21:35:21.143525+0000","last_clean":"2024-03-26T21:35:21.143525+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:35:21.143525+0000","last_undegraded":"2024-03-26T21:35:21.143525+0000","last_fullsized":"2024-03-26T21:35:21.143525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:36.311788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:36.311788+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121169+0000","last_change":"2024-03-26T21:32:23.910111+0000","last_active":"2024-03-26T21:35:21.121169+0000","last_peered":"2024-03-26T21:35:21.121169+0000","last_clean":"2024-03-26T21:35:21.121169+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:35:21.121169+0000","last_undegraded":"2024-03-26T21:35:21.121169+0000","last_fullsized":"2024-03-26T21:35:21.121169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.909981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.909981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"146'13","reported_seq":283,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078185+0000","last_change":"2024-03-26T21:35:09.889342+0000","last_active":"2024-03-26T21:35:21.078185+0000","last_peered":"2024-03-26T21:35:21.078185+0000","last_clean":"2024-03-26T21:35:21.078185+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:35:21.078185+0000","last_undegraded":"2024-03-26T21:35:21.078185+0000","last_fullsized":"2024-03-26T21:35:21.078185+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:35:09.889159+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.889159+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.850380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":256,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:24.088854+0000","last_change":"2024-03-26T21:35:24.088893+0000","last_active":"2024-03-26T21:35:24.088854+0000","last_peered":"2024-03-26T21:35:24.088854+0000","last_clean":"2024-03-26T21:35:24.088854+0000","last_became_active":"2024-03-26T21:32:55.468099+0000","last_became_peered":"2024-03-26T21:32:55.468099+0000","last_unstale":"2024-03-26T21:35:24.088854+0000","last_undegraded":"2024-03-26T21:35:24.088854+0000","last_fullsized":"2024-03-26T21:35:24.088854+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:24.088758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:24.088758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:35.506081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":243,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142910+0000","last_change":"2024-03-26T21:32:23.286349+0000","last_active":"2024-03-26T21:35:21.142910+0000","last_peered":"2024-03-26T21:35:21.142910+0000","last_clean":"2024-03-26T21:35:21.142910+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:35:21.142910+0000","last_undegraded":"2024-03-26T21:35:21.142910+0000","last_fullsized":"2024-03-26T21:35:21.142910+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:23.286238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:23.286238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.539911+0000","last_change":"2024-03-26T21:35:21.539939+0000","last_active":"2024-03-26T21:35:21.539911+0000","last_peered":"2024-03-26T21:35:21.539911+0000","last_clean":"2024-03-26T21:35:21.539911+0000","last_became_active":"2024-03-26T21:32:55.470296+0000","last_became_peered":"2024-03-26T21:32:55.470296+0000","last_unstale":"2024-03-26T21:35:21.539911+0000","last_undegraded":"2024-03-26T21:35:21.539911+0000","last_fullsized":"2024-03-26T21:35:21.539911+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:21.539821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:21.539821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.230060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":275,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121967+0000","last_change":"2024-03-26T21:35:08.078326+0000","last_active":"2024-03-26T21:35:21.121967+0000","last_peered":"2024-03-26T21:35:21.121967+0000","last_clean":"2024-03-26T21:35:21.121967+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:35:21.121967+0000","last_undegraded":"2024-03-26T21:35:21.121967+0000","last_fullsized":"2024-03-26T21:35:21.121967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:35:08.078238+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.078238+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.504484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080856+0000","last_change":"2024-03-26T21:33:27.579541+0000","last_active":"2024-03-26T21:35:21.080856+0000","last_peered":"2024-03-26T21:35:21.080856+0000","last_clean":"2024-03-26T21:35:21.080856+0000","last_became_active":"2024-03-26T21:32:55.470290+0000","last_became_peered":"2024-03-26T21:32:55.470290+0000","last_unstale":"2024-03-26T21:35:21.080856+0000","last_undegraded":"2024-03-26T21:35:21.080856+0000","last_fullsized":"2024-03-26T21:35:21.080856+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:27.579465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:33:27.579465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.076733+0000","last_change":"2024-03-26T21:32:41.663300+0000","last_active":"2024-03-26T21:35:21.076733+0000","last_peered":"2024-03-26T21:35:21.076733+0000","last_clean":"2024-03-26T21:35:21.076733+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:35:21.076733+0000","last_undegraded":"2024-03-26T21:35:21.076733+0000","last_fullsized":"2024-03-26T21:35:21.076733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:41.663183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:41.663183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.076594+0000","last_change":"2024-03-26T21:35:12.106142+0000","last_active":"2024-03-26T21:35:21.076594+0000","last_peered":"2024-03-26T21:35:21.076594+0000","last_clean":"2024-03-26T21:35:21.076594+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:35:21.076594+0000","last_undegraded":"2024-03-26T21:35:21.076594+0000","last_fullsized":"2024-03-26T21:35:21.076594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:12.106061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.106061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:38.680498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":270,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142695+0000","last_change":"2024-03-26T21:32:18.220547+0000","last_active":"2024-03-26T21:35:21.142695+0000","last_peered":"2024-03-26T21:35:21.142695+0000","last_clean":"2024-03-26T21:35:21.142695+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:35:21.142695+0000","last_undegraded":"2024-03-26T21:35:21.142695+0000","last_fullsized":"2024-03-26T21:35:21.142695+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:32:18.214216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:18.214216+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"178'2393","reported_seq":2413,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.326831+0000","last_change":"2024-03-26T21:34:58.693347+0000","last_active":"2024-03-26T21:35:26.326831+0000","last_peered":"2024-03-26T21:35:26.326831+0000","last_clean":"2024-03-26T21:35:26.326831+0000","last_became_active":"2024-03-26T21:34:58.692956+0000","last_became_peered":"2024-03-26T21:34:58.692956+0000","last_unstale":"2024-03-26T21:35:26.326831+0000","last_undegraded":"2024-03-26T21:35:26.326831+0000","last_fullsized":"2024-03-26T21:35:26.326831+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2393,"log_dups_size":0,"ondisk_log_size":2393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.842288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156827648,"num_objects":2393,"num_object_clones":0,"num_object_copies":7179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":153152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.079374+0000","last_change":"2024-03-26T21:35:19.938802+0000","last_active":"2024-03-26T21:35:21.079374+0000","last_peered":"2024-03-26T21:35:21.079374+0000","last_clean":"2024-03-26T21:35:21.079374+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:35:21.079374+0000","last_undegraded":"2024-03-26T21:35:21.079374+0000","last_fullsized":"2024-03-26T21:35:21.079374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.938637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.938637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.681951+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":279,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081206+0000","last_change":"2024-03-26T21:34:28.712452+0000","last_active":"2024-03-26T21:35:21.081206+0000","last_peered":"2024-03-26T21:35:21.081206+0000","last_clean":"2024-03-26T21:35:21.081206+0000","last_became_active":"2024-03-26T21:32:55.475860+0000","last_became_peered":"2024-03-26T21:32:55.475860+0000","last_unstale":"2024-03-26T21:35:21.081206+0000","last_undegraded":"2024-03-26T21:35:21.081206+0000","last_fullsized":"2024-03-26T21:35:21.081206+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:34:28.712368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.712368+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.654006+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":281,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078825+0000","last_change":"2024-03-26T21:34:32.761556+0000","last_active":"2024-03-26T21:35:21.078825+0000","last_peered":"2024-03-26T21:35:21.078825+0000","last_clean":"2024-03-26T21:35:21.078825+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:35:21.078825+0000","last_undegraded":"2024-03-26T21:35:21.078825+0000","last_fullsized":"2024-03-26T21:35:21.078825+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:32.761425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.761425+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.953919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"134'10","reported_seq":346,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080370+0000","last_change":"2024-03-26T21:34:30.686544+0000","last_active":"2024-03-26T21:35:21.080370+0000","last_peered":"2024-03-26T21:35:21.080370+0000","last_clean":"2024-03-26T21:35:21.080370+0000","last_became_active":"2024-03-26T21:32:55.471127+0000","last_became_peered":"2024-03-26T21:32:55.471127+0000","last_unstale":"2024-03-26T21:35:21.080370+0000","last_undegraded":"2024-03-26T21:35:21.080370+0000","last_fullsized":"2024-03-26T21:35:21.080370+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:30.686411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.686411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.859040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"178'2359","reported_seq":2379,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.370263+0000","last_change":"2024-03-26T21:34:59.372364+0000","last_active":"2024-03-26T21:35:26.370263+0000","last_peered":"2024-03-26T21:35:26.370263+0000","last_clean":"2024-03-26T21:35:26.370263+0000","last_became_active":"2024-03-26T21:34:59.371915+0000","last_became_peered":"2024-03-26T21:34:59.371915+0000","last_unstale":"2024-03-26T21:35:26.370263+0000","last_undegraded":"2024-03-26T21:35:26.370263+0000","last_fullsized":"2024-03-26T21:35:26.370263+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2359,"log_dups_size":0,"ondisk_log_size":2359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:08.085476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154599424,"num_objects":2359,"num_object_clones":0,"num_object_copies":7077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2359,"num_write_kb":150976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":240,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.146376+0000","last_change":"2024-03-26T21:32:42.225753+0000","last_active":"2024-03-26T21:35:21.146376+0000","last_peered":"2024-03-26T21:35:21.146376+0000","last_clean":"2024-03-26T21:35:21.146376+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:35:21.146376+0000","last_undegraded":"2024-03-26T21:35:21.146376+0000","last_fullsized":"2024-03-26T21:35:21.146376+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:42.225645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:42.225645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":276,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081148+0000","last_change":"2024-03-26T21:35:15.464143+0000","last_active":"2024-03-26T21:35:21.081148+0000","last_peered":"2024-03-26T21:35:21.081148+0000","last_clean":"2024-03-26T21:35:21.081148+0000","last_became_active":"2024-03-26T21:32:55.471766+0000","last_became_peered":"2024-03-26T21:32:55.471766+0000","last_unstale":"2024-03-26T21:35:21.081148+0000","last_undegraded":"2024-03-26T21:35:21.081148+0000","last_fullsized":"2024-03-26T21:35:21.081148+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:35:15.464062+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.464062+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:29.495127+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":191,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127669+0000","last_change":"2024-03-26T21:34:18.762352+0000","last_active":"2024-03-26T21:35:28.127669+0000","last_peered":"2024-03-26T21:35:28.127669+0000","last_clean":"2024-03-26T21:35:28.127669+0000","last_became_active":"2024-03-26T21:32:55.467883+0000","last_became_peered":"2024-03-26T21:32:55.467883+0000","last_unstale":"2024-03-26T21:35:28.127669+0000","last_undegraded":"2024-03-26T21:35:28.127669+0000","last_fullsized":"2024-03-26T21:35:28.127669+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:18.762259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.762259+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.929171+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082660+0000","last_change":"2024-03-26T21:34:20.747940+0000","last_active":"2024-03-26T21:35:21.082660+0000","last_peered":"2024-03-26T21:35:21.082660+0000","last_clean":"2024-03-26T21:35:21.082660+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:35:21.082660+0000","last_undegraded":"2024-03-26T21:35:21.082660+0000","last_fullsized":"2024-03-26T21:35:21.082660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.747694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.747694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.007716+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"178'2508","reported_seq":2528,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.927556+0000","last_change":"2024-03-26T21:34:59.371731+0000","last_active":"2024-03-26T21:35:26.927556+0000","last_peered":"2024-03-26T21:35:26.927556+0000","last_clean":"2024-03-26T21:35:26.927556+0000","last_became_active":"2024-03-26T21:34:59.371417+0000","last_became_peered":"2024-03-26T21:34:59.371417+0000","last_unstale":"2024-03-26T21:35:26.927556+0000","last_undegraded":"2024-03-26T21:35:26.927556+0000","last_fullsized":"2024-03-26T21:35:26.927556+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:12.074404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":160512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080668+0000","last_change":"2024-03-26T21:34:08.816123+0000","last_active":"2024-03-26T21:35:21.080668+0000","last_peered":"2024-03-26T21:35:21.080668+0000","last_clean":"2024-03-26T21:35:21.080668+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:35:21.080668+0000","last_undegraded":"2024-03-26T21:35:21.080668+0000","last_fullsized":"2024-03-26T21:35:21.080668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.816046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.816046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.663133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:23.046518+0000","last_change":"2024-03-26T21:35:23.046553+0000","last_active":"2024-03-26T21:35:23.046518+0000","last_peered":"2024-03-26T21:35:23.046518+0000","last_clean":"2024-03-26T21:35:23.046518+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:35:23.046518+0000","last_undegraded":"2024-03-26T21:35:23.046518+0000","last_fullsized":"2024-03-26T21:35:23.046518+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:23.046435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:23.046435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:35.412387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081206+0000","last_change":"2024-03-26T21:34:26.757750+0000","last_active":"2024-03-26T21:35:21.081206+0000","last_peered":"2024-03-26T21:35:21.081206+0000","last_clean":"2024-03-26T21:35:21.081206+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:35:21.081206+0000","last_undegraded":"2024-03-26T21:35:21.081206+0000","last_fullsized":"2024-03-26T21:35:21.081206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.757669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.757669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:56.675099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":304,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077709+0000","last_change":"2024-03-26T21:35:09.151841+0000","last_active":"2024-03-26T21:35:21.077709+0000","last_peered":"2024-03-26T21:35:21.077709+0000","last_clean":"2024-03-26T21:35:21.077709+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:35:21.077709+0000","last_undegraded":"2024-03-26T21:35:21.077709+0000","last_fullsized":"2024-03-26T21:35:21.077709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:35:09.151698+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.151698+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.490492+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"178'2498","reported_seq":2518,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.585139+0000","last_change":"2024-03-26T21:34:58.694137+0000","last_active":"2024-03-26T21:35:26.585139+0000","last_peered":"2024-03-26T21:35:26.585139+0000","last_clean":"2024-03-26T21:35:26.585139+0000","last_became_active":"2024-03-26T21:34:58.693834+0000","last_became_peered":"2024-03-26T21:34:58.693834+0000","last_unstale":"2024-03-26T21:35:26.585139+0000","last_undegraded":"2024-03-26T21:35:26.585139+0000","last_fullsized":"2024-03-26T21:35:26.585139+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.096424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163708928,"num_objects":2498,"num_object_clones":0,"num_object_copies":7494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2498,"num_write_kb":159872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.120833+0000","last_change":"2024-03-26T21:35:13.040636+0000","last_active":"2024-03-26T21:35:21.120833+0000","last_peered":"2024-03-26T21:35:21.120833+0000","last_clean":"2024-03-26T21:35:21.120833+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:35:21.120833+0000","last_undegraded":"2024-03-26T21:35:21.120833+0000","last_fullsized":"2024-03-26T21:35:21.120833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.040364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.040364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.367879+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082676+0000","last_change":"2024-03-26T21:35:18.075830+0000","last_active":"2024-03-26T21:35:21.082676+0000","last_peered":"2024-03-26T21:35:21.082676+0000","last_clean":"2024-03-26T21:35:21.082676+0000","last_became_active":"2024-03-26T21:32:55.471707+0000","last_became_peered":"2024-03-26T21:32:55.471707+0000","last_unstale":"2024-03-26T21:35:21.082676+0000","last_undegraded":"2024-03-26T21:35:21.082676+0000","last_fullsized":"2024-03-26T21:35:21.082676+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:18.075661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:18.075661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:31.784375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077138+0000","last_change":"2024-03-26T21:34:14.695183+0000","last_active":"2024-03-26T21:35:21.077138+0000","last_peered":"2024-03-26T21:35:21.077138+0000","last_clean":"2024-03-26T21:35:21.077138+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:35:21.077138+0000","last_undegraded":"2024-03-26T21:35:21.077138+0000","last_fullsized":"2024-03-26T21:35:21.077138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.695081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.695081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121499+0000","last_change":"2024-03-26T21:34:15.948871+0000","last_active":"2024-03-26T21:35:21.121499+0000","last_peered":"2024-03-26T21:35:21.121499+0000","last_clean":"2024-03-26T21:35:21.121499+0000","last_became_active":"2024-03-26T21:32:55.478663+0000","last_became_peered":"2024-03-26T21:32:55.478663+0000","last_unstale":"2024-03-26T21:35:21.121499+0000","last_undegraded":"2024-03-26T21:35:21.121499+0000","last_fullsized":"2024-03-26T21:35:21.121499+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:15.948730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.948730+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"178'2499","reported_seq":2519,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.345901+0000","last_change":"2024-03-26T21:34:59.371815+0000","last_active":"2024-03-26T21:35:26.345901+0000","last_peered":"2024-03-26T21:35:26.345901+0000","last_clean":"2024-03-26T21:35:26.345901+0000","last_became_active":"2024-03-26T21:34:59.371434+0000","last_became_peered":"2024-03-26T21:34:59.371434+0000","last_unstale":"2024-03-26T21:35:26.345901+0000","last_undegraded":"2024-03-26T21:35:26.345901+0000","last_fullsized":"2024-03-26T21:35:26.345901+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.142848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163774464,"num_objects":2499,"num_object_clones":0,"num_object_copies":7497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":159936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":249,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142455+0000","last_change":"2024-03-26T21:34:10.292343+0000","last_active":"2024-03-26T21:35:21.142455+0000","last_peered":"2024-03-26T21:35:21.142455+0000","last_clean":"2024-03-26T21:35:21.142455+0000","last_became_active":"2024-03-26T21:32:55.468927+0000","last_became_peered":"2024-03-26T21:32:55.468927+0000","last_unstale":"2024-03-26T21:35:21.142455+0000","last_undegraded":"2024-03-26T21:35:21.142455+0000","last_fullsized":"2024-03-26T21:35:21.142455+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:10.292229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.292229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":269,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.079068+0000","last_change":"2024-03-26T21:34:16.662556+0000","last_active":"2024-03-26T21:35:21.079068+0000","last_peered":"2024-03-26T21:35:21.079068+0000","last_clean":"2024-03-26T21:35:21.079068+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:35:21.079068+0000","last_undegraded":"2024-03-26T21:35:21.079068+0000","last_fullsized":"2024-03-26T21:35:21.079068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:34:16.662471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:16.662471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.912431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":178,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127319+0000","last_change":"2024-03-26T21:35:12.163203+0000","last_active":"2024-03-26T21:35:28.127319+0000","last_peered":"2024-03-26T21:35:28.127319+0000","last_clean":"2024-03-26T21:35:28.127319+0000","last_became_active":"2024-03-26T21:32:55.480439+0000","last_became_peered":"2024-03-26T21:32:55.480439+0000","last_unstale":"2024-03-26T21:35:28.127319+0000","last_undegraded":"2024-03-26T21:35:28.127319+0000","last_fullsized":"2024-03-26T21:35:28.127319+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:12.163075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.163075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.583097+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":341,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.076812+0000","last_change":"2024-03-26T21:34:13.674583+0000","last_active":"2024-03-26T21:35:21.076812+0000","last_peered":"2024-03-26T21:35:21.076812+0000","last_clean":"2024-03-26T21:35:21.076812+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:35:21.076812+0000","last_undegraded":"2024-03-26T21:35:21.076812+0000","last_fullsized":"2024-03-26T21:35:21.076812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:34:13.674406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.674406+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121283+0000","last_change":"2024-03-26T21:34:18.955506+0000","last_active":"2024-03-26T21:35:21.121283+0000","last_peered":"2024-03-26T21:35:21.121283+0000","last_clean":"2024-03-26T21:35:21.121283+0000","last_became_active":"2024-03-26T21:32:55.478576+0000","last_became_peered":"2024-03-26T21:32:55.478576+0000","last_unstale":"2024-03-26T21:35:21.121283+0000","last_undegraded":"2024-03-26T21:35:21.121283+0000","last_fullsized":"2024-03-26T21:35:21.121283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:18.955400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.955400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.675127+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"178'2468","reported_seq":2488,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:27.022264+0000","last_change":"2024-03-26T21:34:58.692851+0000","last_active":"2024-03-26T21:35:27.022264+0000","last_peered":"2024-03-26T21:35:27.022264+0000","last_clean":"2024-03-26T21:35:27.022264+0000","last_became_active":"2024-03-26T21:34:58.692166+0000","last_became_peered":"2024-03-26T21:34:58.692166+0000","last_unstale":"2024-03-26T21:35:27.022264+0000","last_undegraded":"2024-03-26T21:35:27.022264+0000","last_fullsized":"2024-03-26T21:35:27.022264+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:01.583848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":157952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.081317+0000","last_change":"2024-03-26T21:33:28.620552+0000","last_active":"2024-03-26T21:35:21.081317+0000","last_peered":"2024-03-26T21:35:21.081317+0000","last_clean":"2024-03-26T21:35:21.081317+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:35:21.081317+0000","last_undegraded":"2024-03-26T21:35:21.081317+0000","last_fullsized":"2024-03-26T21:35:21.081317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:28.620444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:33:28.620444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144043+0000","last_change":"2024-03-26T21:34:24.599907+0000","last_active":"2024-03-26T21:35:21.144043+0000","last_peered":"2024-03-26T21:35:21.144043+0000","last_clean":"2024-03-26T21:35:21.144043+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:35:21.144043+0000","last_undegraded":"2024-03-26T21:35:21.144043+0000","last_fullsized":"2024-03-26T21:35:21.144043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.599655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.967423+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082747+0000","last_change":"2024-03-26T21:34:29.929261+0000","last_active":"2024-03-26T21:35:21.082747+0000","last_peered":"2024-03-26T21:35:21.082747+0000","last_clean":"2024-03-26T21:35:21.082747+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:35:21.082747+0000","last_undegraded":"2024-03-26T21:35:21.082747+0000","last_fullsized":"2024-03-26T21:35:21.082747+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:34:29.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:39.201699+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":288,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144758+0000","last_change":"2024-03-26T21:34:10.523681+0000","last_active":"2024-03-26T21:35:21.144758+0000","last_peered":"2024-03-26T21:35:21.144758+0000","last_clean":"2024-03-26T21:35:21.144758+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:35:21.144758+0000","last_undegraded":"2024-03-26T21:35:21.144758+0000","last_fullsized":"2024-03-26T21:35:21.144758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:34:10.523547+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.523547+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.692847+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.141412+0000","last_change":"2024-03-26T21:34:32.868281+0000","last_active":"2024-03-26T21:35:21.141412+0000","last_peered":"2024-03-26T21:35:21.141412+0000","last_clean":"2024-03-26T21:35:21.141412+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:35:21.141412+0000","last_undegraded":"2024-03-26T21:35:21.141412+0000","last_fullsized":"2024-03-26T21:35:21.141412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.868200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.530052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"178'2521","reported_seq":2541,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.367818+0000","last_change":"2024-03-26T21:34:58.687897+0000","last_active":"2024-03-26T21:35:26.367818+0000","last_peered":"2024-03-26T21:35:26.367818+0000","last_clean":"2024-03-26T21:35:26.367818+0000","last_became_active":"2024-03-26T21:34:58.687512+0000","last_became_peered":"2024-03-26T21:34:58.687512+0000","last_unstale":"2024-03-26T21:35:26.367818+0000","last_undegraded":"2024-03-26T21:35:26.367818+0000","last_fullsized":"2024-03-26T21:35:26.367818+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2521,"log_dups_size":0,"ondisk_log_size":2521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:11.604741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2521,"num_write_kb":161344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":249,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077008+0000","last_change":"2024-03-26T21:33:26.736579+0000","last_active":"2024-03-26T21:35:21.077008+0000","last_peered":"2024-03-26T21:35:21.077008+0000","last_clean":"2024-03-26T21:35:21.077008+0000","last_became_active":"2024-03-26T21:32:55.474113+0000","last_became_peered":"2024-03-26T21:32:55.474113+0000","last_unstale":"2024-03-26T21:35:21.077008+0000","last_undegraded":"2024-03-26T21:35:21.077008+0000","last_fullsized":"2024-03-26T21:35:21.077008+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:26.736456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:26.736456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":512,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080916+0000","last_change":"2024-03-26T21:34:17.683647+0000","last_active":"2024-03-26T21:35:21.080916+0000","last_peered":"2024-03-26T21:35:21.080916+0000","last_clean":"2024-03-26T21:35:21.080916+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:35:21.080916+0000","last_undegraded":"2024-03-26T21:35:21.080916+0000","last_fullsized":"2024-03-26T21:35:21.080916+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:34:17.683557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:34:17.683557+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.639537+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.074898+0000","last_change":"2024-03-26T21:34:28.378904+0000","last_active":"2024-03-26T21:35:21.074898+0000","last_peered":"2024-03-26T21:35:21.074898+0000","last_clean":"2024-03-26T21:35:21.074898+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:35:21.074898+0000","last_undegraded":"2024-03-26T21:35:21.074898+0000","last_fullsized":"2024-03-26T21:35:21.074898+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:28.378787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.378787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:34.024112+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"134'26","reported_seq":380,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078710+0000","last_change":"2024-03-26T21:34:31.976293+0000","last_active":"2024-03-26T21:35:21.078710+0000","last_peered":"2024-03-26T21:35:21.078710+0000","last_clean":"2024-03-26T21:35:21.078710+0000","last_became_active":"2024-03-26T21:32:55.477984+0000","last_became_peered":"2024-03-26T21:32:55.477984+0000","last_unstale":"2024-03-26T21:35:21.078710+0000","last_undegraded":"2024-03-26T21:35:21.078710+0000","last_fullsized":"2024-03-26T21:35:21.078710+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'26","last_scrub_stamp":"2024-03-26T21:34:31.976190+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.976190+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.446068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.082522+0000","last_change":"2024-03-26T21:34:20.660945+0000","last_active":"2024-03-26T21:35:21.082522+0000","last_peered":"2024-03-26T21:35:21.082522+0000","last_clean":"2024-03-26T21:35:21.082522+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:35:21.082522+0000","last_undegraded":"2024-03-26T21:35:21.082522+0000","last_fullsized":"2024-03-26T21:35:21.082522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.660825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.660825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.066534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"178'2495","reported_seq":2515,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.605606+0000","last_change":"2024-03-26T21:34:58.693396+0000","last_active":"2024-03-26T21:35:26.605606+0000","last_peered":"2024-03-26T21:35:26.605606+0000","last_clean":"2024-03-26T21:35:26.605606+0000","last_became_active":"2024-03-26T21:34:58.693101+0000","last_became_peered":"2024-03-26T21:34:58.693101+0000","last_unstale":"2024-03-26T21:35:26.605606+0000","last_undegraded":"2024-03-26T21:35:26.605606+0000","last_fullsized":"2024-03-26T21:35:26.605606+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:17.317391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":159680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144540+0000","last_change":"2024-03-26T21:34:14.558500+0000","last_active":"2024-03-26T21:35:21.144540+0000","last_peered":"2024-03-26T21:35:21.144540+0000","last_clean":"2024-03-26T21:35:21.144540+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:35:21.144540+0000","last_undegraded":"2024-03-26T21:35:21.144540+0000","last_fullsized":"2024-03-26T21:35:21.144540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.557934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.557934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.351356+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.073607+0000","last_change":"2024-03-26T21:34:25.034122+0000","last_active":"2024-03-26T21:35:21.073607+0000","last_peered":"2024-03-26T21:35:21.073607+0000","last_clean":"2024-03-26T21:35:21.073607+0000","last_became_active":"2024-03-26T21:32:55.475773+0000","last_became_peered":"2024-03-26T21:32:55.475773+0000","last_unstale":"2024-03-26T21:35:21.073607+0000","last_undegraded":"2024-03-26T21:35:21.073607+0000","last_fullsized":"2024-03-26T21:35:21.073607+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.033987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.033987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:25.283140+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":683,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:24.884482+0000","last_change":"2024-03-26T21:35:11.460682+0000","last_active":"2024-03-26T21:35:24.884482+0000","last_peered":"2024-03-26T21:35:24.884482+0000","last_clean":"2024-03-26T21:35:24.884482+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:35:24.884482+0000","last_undegraded":"2024-03-26T21:35:24.884482+0000","last_fullsized":"2024-03-26T21:35:24.884482+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:35:11.460547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.460547+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.085903+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":294,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121680+0000","last_change":"2024-03-26T21:32:39.859396+0000","last_active":"2024-03-26T21:35:21.121680+0000","last_peered":"2024-03-26T21:35:21.121680+0000","last_clean":"2024-03-26T21:35:21.121680+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:35:21.121680+0000","last_undegraded":"2024-03-26T21:35:21.121680+0000","last_fullsized":"2024-03-26T21:35:21.121680+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:32:39.858918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:39.858918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077461+0000","last_change":"2024-03-26T21:34:09.662284+0000","last_active":"2024-03-26T21:35:21.077461+0000","last_peered":"2024-03-26T21:35:21.077461+0000","last_clean":"2024-03-26T21:35:21.077461+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:35:21.077461+0000","last_undegraded":"2024-03-26T21:35:21.077461+0000","last_fullsized":"2024-03-26T21:35:21.077461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.662190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.662190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.442388+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"178'2520","reported_seq":2537,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:27.015967+0000","last_change":"2024-03-26T21:34:58.693123+0000","last_active":"2024-03-26T21:35:27.015967+0000","last_peered":"2024-03-26T21:35:27.015967+0000","last_clean":"2024-03-26T21:35:27.015967+0000","last_became_active":"2024-03-26T21:34:58.692159+0000","last_became_peered":"2024-03-26T21:34:58.692159+0000","last_unstale":"2024-03-26T21:35:27.015967+0000","last_undegraded":"2024-03-26T21:35:27.015967+0000","last_fullsized":"2024-03-26T21:35:27.015967+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2520,"log_dups_size":0,"ondisk_log_size":2520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:08.426555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":161280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144489+0000","last_change":"2024-03-26T21:34:08.527352+0000","last_active":"2024-03-26T21:35:21.144489+0000","last_peered":"2024-03-26T21:35:21.144489+0000","last_clean":"2024-03-26T21:35:21.144489+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:35:21.144489+0000","last_undegraded":"2024-03-26T21:35:21.144489+0000","last_fullsized":"2024-03-26T21:35:21.144489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.527230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.527230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.546653+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144675+0000","last_change":"2024-03-26T21:35:17.424167+0000","last_active":"2024-03-26T21:35:21.144675+0000","last_peered":"2024-03-26T21:35:21.144675+0000","last_clean":"2024-03-26T21:35:21.144675+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:35:21.144675+0000","last_undegraded":"2024-03-26T21:35:21.144675+0000","last_fullsized":"2024-03-26T21:35:21.144675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:17.424050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:35:17.424050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:39.574813+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":292,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121350+0000","last_change":"2024-03-26T21:35:11.003728+0000","last_active":"2024-03-26T21:35:21.121350+0000","last_peered":"2024-03-26T21:35:21.121350+0000","last_clean":"2024-03-26T21:35:21.121350+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:35:21.121350+0000","last_undegraded":"2024-03-26T21:35:21.121350+0000","last_fullsized":"2024-03-26T21:35:21.121350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:35:11.003580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.003580+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.140682+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077798+0000","last_change":"2024-03-26T21:35:15.119981+0000","last_active":"2024-03-26T21:35:21.077798+0000","last_peered":"2024-03-26T21:35:21.077798+0000","last_clean":"2024-03-26T21:35:21.077798+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:35:21.077798+0000","last_undegraded":"2024-03-26T21:35:21.077798+0000","last_fullsized":"2024-03-26T21:35:21.077798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.119862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.119862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.753214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"178'2453","reported_seq":2473,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.854279+0000","last_change":"2024-03-26T21:34:58.689609+0000","last_active":"2024-03-26T21:35:26.854279+0000","last_peered":"2024-03-26T21:35:26.854279+0000","last_clean":"2024-03-26T21:35:26.854279+0000","last_became_active":"2024-03-26T21:34:58.689106+0000","last_became_peered":"2024-03-26T21:34:58.689106+0000","last_unstale":"2024-03-26T21:35:26.854279+0000","last_undegraded":"2024-03-26T21:35:26.854279+0000","last_fullsized":"2024-03-26T21:35:26.854279+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2453,"log_dups_size":0,"ondisk_log_size":2453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.882021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160759808,"num_objects":2453,"num_object_clones":0,"num_object_copies":7359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":156992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":240,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.072816+0000","last_change":"2024-03-26T21:32:13.010288+0000","last_active":"2024-03-26T21:35:21.072816+0000","last_peered":"2024-03-26T21:35:21.072816+0000","last_clean":"2024-03-26T21:35:21.072816+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:35:21.072816+0000","last_undegraded":"2024-03-26T21:35:21.072816+0000","last_fullsized":"2024-03-26T21:35:21.072816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:13.010170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:13.010170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:25.841765+0000","last_change":"2024-03-26T21:35:25.841787+0000","last_active":"2024-03-26T21:35:25.841765+0000","last_peered":"2024-03-26T21:35:25.841765+0000","last_clean":"2024-03-26T21:35:25.841765+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:35:25.841765+0000","last_undegraded":"2024-03-26T21:35:25.841765+0000","last_fullsized":"2024-03-26T21:35:25.841765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.841708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.841708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:45.777516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":280,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144249+0000","last_change":"2024-03-26T21:35:15.509109+0000","last_active":"2024-03-26T21:35:21.144249+0000","last_peered":"2024-03-26T21:35:21.144249+0000","last_clean":"2024-03-26T21:35:21.144249+0000","last_became_active":"2024-03-26T21:32:55.465412+0000","last_became_peered":"2024-03-26T21:32:55.465412+0000","last_unstale":"2024-03-26T21:35:21.144249+0000","last_undegraded":"2024-03-26T21:35:21.144249+0000","last_fullsized":"2024-03-26T21:35:21.144249+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:35:15.508751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.508751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.773583+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":335,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121290+0000","last_change":"2024-03-26T21:32:11.923454+0000","last_active":"2024-03-26T21:35:21.121290+0000","last_peered":"2024-03-26T21:35:21.121290+0000","last_clean":"2024-03-26T21:35:21.121290+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:35:21.121290+0000","last_undegraded":"2024-03-26T21:35:21.121290+0000","last_fullsized":"2024-03-26T21:35:21.121290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:32:11.916784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:32:11.916784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"178'2379","reported_seq":2398,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.359602+0000","last_change":"2024-03-26T21:34:59.372041+0000","last_active":"2024-03-26T21:35:26.359602+0000","last_peered":"2024-03-26T21:35:26.359602+0000","last_clean":"2024-03-26T21:35:26.359602+0000","last_became_active":"2024-03-26T21:34:59.371634+0000","last_became_peered":"2024-03-26T21:34:59.371634+0000","last_unstale":"2024-03-26T21:35:26.359602+0000","last_undegraded":"2024-03-26T21:35:26.359602+0000","last_fullsized":"2024-03-26T21:35:26.359602+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2379,"log_dups_size":0,"ondisk_log_size":2379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:21.299560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":7137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":152256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":256,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.080789+0000","last_change":"2024-03-26T21:35:10.438550+0000","last_active":"2024-03-26T21:35:21.080789+0000","last_peered":"2024-03-26T21:35:21.080789+0000","last_clean":"2024-03-26T21:35:21.080789+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:35:21.080789+0000","last_undegraded":"2024-03-26T21:35:21.080789+0000","last_fullsized":"2024-03-26T21:35:21.080789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:10.438407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:10.438407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.156978+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":254,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.121808+0000","last_change":"2024-03-26T21:34:10.057922+0000","last_active":"2024-03-26T21:35:21.121808+0000","last_peered":"2024-03-26T21:35:21.121808+0000","last_clean":"2024-03-26T21:35:21.121808+0000","last_became_active":"2024-03-26T21:32:55.479274+0000","last_became_peered":"2024-03-26T21:32:55.479274+0000","last_unstale":"2024-03-26T21:35:21.121808+0000","last_undegraded":"2024-03-26T21:35:21.121808+0000","last_fullsized":"2024-03-26T21:35:21.121808+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:34:10.057797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.057797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:23.442880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":195,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127429+0000","last_change":"2024-03-26T21:35:26.141597+0000","last_active":"2024-03-26T21:35:28.127429+0000","last_peered":"2024-03-26T21:35:28.127429+0000","last_clean":"2024-03-26T21:35:28.127429+0000","last_became_active":"2024-03-26T21:32:55.469472+0000","last_became_peered":"2024-03-26T21:32:55.469472+0000","last_unstale":"2024-03-26T21:35:28.127429+0000","last_undegraded":"2024-03-26T21:35:28.127429+0000","last_fullsized":"2024-03-26T21:35:28.127429+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:35:26.141418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:26.141418+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:49.710634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"134'10","reported_seq":346,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143858+0000","last_change":"2024-03-26T21:34:12.341709+0000","last_active":"2024-03-26T21:35:21.143858+0000","last_peered":"2024-03-26T21:35:21.143858+0000","last_clean":"2024-03-26T21:35:21.143858+0000","last_became_active":"2024-03-26T21:32:55.476581+0000","last_became_peered":"2024-03-26T21:32:55.476581+0000","last_unstale":"2024-03-26T21:35:21.143858+0000","last_undegraded":"2024-03-26T21:35:21.143858+0000","last_fullsized":"2024-03-26T21:35:21.143858+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:12.341310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.341310+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:33.217651+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"178'2472","reported_seq":2491,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.850379+0000","last_change":"2024-03-26T21:34:59.370935+0000","last_active":"2024-03-26T21:35:26.850379+0000","last_peered":"2024-03-26T21:35:26.850379+0000","last_clean":"2024-03-26T21:35:26.850379+0000","last_became_active":"2024-03-26T21:34:59.370503+0000","last_became_peered":"2024-03-26T21:34:59.370503+0000","last_unstale":"2024-03-26T21:35:26.850379+0000","last_undegraded":"2024-03-26T21:35:26.850379+0000","last_fullsized":"2024-03-26T21:35:26.850379+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2472,"log_dups_size":0,"ondisk_log_size":2472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:21.453666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2472,"num_write_kb":158208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":160,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127671+0000","last_change":"2024-03-26T21:34:33.905393+0000","last_active":"2024-03-26T21:35:28.127671+0000","last_peered":"2024-03-26T21:35:28.127671+0000","last_clean":"2024-03-26T21:35:28.127671+0000","last_became_active":"2024-03-26T21:32:55.476227+0000","last_became_peered":"2024-03-26T21:32:55.476227+0000","last_unstale":"2024-03-26T21:35:28.127671+0000","last_undegraded":"2024-03-26T21:35:28.127671+0000","last_fullsized":"2024-03-26T21:35:28.127671+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.905276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.905276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.967842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.142595+0000","last_change":"2024-03-26T21:32:17.198343+0000","last_active":"2024-03-26T21:35:21.142595+0000","last_peered":"2024-03-26T21:35:21.142595+0000","last_clean":"2024-03-26T21:35:21.142595+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:35:21.142595+0000","last_undegraded":"2024-03-26T21:35:21.142595+0000","last_fullsized":"2024-03-26T21:35:21.142595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:17.198228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:32:17.198228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":280,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143729+0000","last_change":"2024-03-26T21:32:33.337863+0000","last_active":"2024-03-26T21:35:21.143729+0000","last_peered":"2024-03-26T21:35:21.143729+0000","last_clean":"2024-03-26T21:35:21.143729+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:35:21.143729+0000","last_undegraded":"2024-03-26T21:35:21.143729+0000","last_fullsized":"2024-03-26T21:35:21.143729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:32:33.337581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:33.337581+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":121,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.126821+0000","last_change":"2024-03-26T21:34:07.791992+0000","last_active":"2024-03-26T21:35:28.126821+0000","last_peered":"2024-03-26T21:35:28.126821+0000","last_clean":"2024-03-26T21:35:28.126821+0000","last_became_active":"2024-03-26T21:32:55.468331+0000","last_became_peered":"2024-03-26T21:32:55.468331+0000","last_unstale":"2024-03-26T21:35:28.126821+0000","last_undegraded":"2024-03-26T21:35:28.126821+0000","last_fullsized":"2024-03-26T21:35:28.126821+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.791763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.791763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.911000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"178'2448","reported_seq":2468,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.588588+0000","last_change":"2024-03-26T21:34:58.691128+0000","last_active":"2024-03-26T21:35:26.588588+0000","last_peered":"2024-03-26T21:35:26.588588+0000","last_clean":"2024-03-26T21:35:26.588588+0000","last_became_active":"2024-03-26T21:34:58.690860+0000","last_became_peered":"2024-03-26T21:34:58.690860+0000","last_unstale":"2024-03-26T21:35:26.588588+0000","last_undegraded":"2024-03-26T21:35:26.588588+0000","last_fullsized":"2024-03-26T21:35:26.588588+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2448,"log_dups_size":0,"ondisk_log_size":2448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.930774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160432128,"num_objects":2448,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2448,"num_write_kb":156672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.078333+0000","last_change":"2024-03-26T21:34:15.695258+0000","last_active":"2024-03-26T21:35:21.078333+0000","last_peered":"2024-03-26T21:35:21.078333+0000","last_clean":"2024-03-26T21:35:21.078333+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:35:21.078333+0000","last_undegraded":"2024-03-26T21:35:21.078333+0000","last_fullsized":"2024-03-26T21:35:21.078333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:15.695126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.695126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:24.175611+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"134'10","reported_seq":229,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.129566+0000","last_change":"2024-03-26T21:35:09.138220+0000","last_active":"2024-03-26T21:35:28.129566+0000","last_peered":"2024-03-26T21:35:28.129566+0000","last_clean":"2024-03-26T21:35:28.129566+0000","last_became_active":"2024-03-26T21:32:55.478624+0000","last_became_peered":"2024-03-26T21:32:55.478624+0000","last_unstale":"2024-03-26T21:35:28.129566+0000","last_undegraded":"2024-03-26T21:35:28.129566+0000","last_fullsized":"2024-03-26T21:35:28.129566+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:35:09.138087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.138087+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:20.372198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143626+0000","last_change":"2024-03-26T21:35:14.322330+0000","last_active":"2024-03-26T21:35:21.143626+0000","last_peered":"2024-03-26T21:35:21.143626+0000","last_clean":"2024-03-26T21:35:21.143626+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:35:21.143626+0000","last_undegraded":"2024-03-26T21:35:21.143626+0000","last_fullsized":"2024-03-26T21:35:21.143626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.322126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.322126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:15.039978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":290,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144646+0000","last_change":"2024-03-26T21:34:33.587083+0000","last_active":"2024-03-26T21:35:21.144646+0000","last_peered":"2024-03-26T21:35:21.144646+0000","last_clean":"2024-03-26T21:35:21.144646+0000","last_became_active":"2024-03-26T21:32:55.471751+0000","last_became_peered":"2024-03-26T21:32:55.471751+0000","last_unstale":"2024-03-26T21:35:21.144646+0000","last_undegraded":"2024-03-26T21:35:21.144646+0000","last_fullsized":"2024-03-26T21:35:21.144646+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:33.586983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.586983+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:35.630361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"179'2658","reported_seq":2678,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.477224+0000","last_change":"2024-03-26T21:34:58.687626+0000","last_active":"2024-03-26T21:35:28.477224+0000","last_peered":"2024-03-26T21:35:28.477224+0000","last_clean":"2024-03-26T21:35:28.477224+0000","last_became_active":"2024-03-26T21:34:58.687260+0000","last_became_peered":"2024-03-26T21:34:58.687260+0000","last_unstale":"2024-03-26T21:35:28.477224+0000","last_undegraded":"2024-03-26T21:35:28.477224+0000","last_fullsized":"2024-03-26T21:35:28.477224+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2658,"log_dups_size":0,"ondisk_log_size":2658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.659671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174194688,"num_objects":2658,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2658,"num_write_kb":170112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.143033+0000","last_change":"2024-03-26T21:34:32.360553+0000","last_active":"2024-03-26T21:35:21.143033+0000","last_peered":"2024-03-26T21:35:21.143033+0000","last_clean":"2024-03-26T21:35:21.143033+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:35:21.143033+0000","last_undegraded":"2024-03-26T21:35:21.143033+0000","last_fullsized":"2024-03-26T21:35:21.143033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.360471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.360471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.149069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077410+0000","last_change":"2024-03-26T21:35:08.865267+0000","last_active":"2024-03-26T21:35:21.077410+0000","last_peered":"2024-03-26T21:35:21.077410+0000","last_clean":"2024-03-26T21:35:21.077410+0000","last_became_active":"2024-03-26T21:32:55.465348+0000","last_became_peered":"2024-03-26T21:32:55.465348+0000","last_unstale":"2024-03-26T21:35:21.077410+0000","last_undegraded":"2024-03-26T21:35:21.077410+0000","last_fullsized":"2024-03-26T21:35:21.077410+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:08.865126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.865126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.367221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":172,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.127158+0000","last_change":"2024-03-26T21:34:24.836203+0000","last_active":"2024-03-26T21:35:28.127158+0000","last_peered":"2024-03-26T21:35:28.127158+0000","last_clean":"2024-03-26T21:35:28.127158+0000","last_became_active":"2024-03-26T21:32:55.463332+0000","last_became_peered":"2024-03-26T21:32:55.463332+0000","last_unstale":"2024-03-26T21:35:28.127158+0000","last_undegraded":"2024-03-26T21:35:28.127158+0000","last_fullsized":"2024-03-26T21:35:28.127158+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.835878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.835878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:36.538447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":191,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.129172+0000","last_change":"2024-03-26T21:35:20.104232+0000","last_active":"2024-03-26T21:35:28.129172+0000","last_peered":"2024-03-26T21:35:28.129172+0000","last_clean":"2024-03-26T21:35:28.129172+0000","last_became_active":"2024-03-26T21:32:55.471887+0000","last_became_peered":"2024-03-26T21:32:55.471887+0000","last_unstale":"2024-03-26T21:35:28.129172+0000","last_undegraded":"2024-03-26T21:35:28.129172+0000","last_fullsized":"2024-03-26T21:35:28.129172+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:35:20.104039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:20.104039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:42.567579+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"178'2512","reported_seq":2532,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.596137+0000","last_change":"2024-03-26T21:34:58.693459+0000","last_active":"2024-03-26T21:35:26.596137+0000","last_peered":"2024-03-26T21:35:26.596137+0000","last_clean":"2024-03-26T21:35:26.596137+0000","last_became_active":"2024-03-26T21:34:58.693174+0000","last_became_peered":"2024-03-26T21:34:58.693174+0000","last_unstale":"2024-03-26T21:35:26.596137+0000","last_undegraded":"2024-03-26T21:35:26.596137+0000","last_fullsized":"2024-03-26T21:35:26.596137+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2512,"log_dups_size":0,"ondisk_log_size":2512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:03.413097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626432,"num_objects":2512,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":160768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.141626+0000","last_change":"2024-03-26T21:34:12.660667+0000","last_active":"2024-03-26T21:35:21.141626+0000","last_peered":"2024-03-26T21:35:21.141626+0000","last_clean":"2024-03-26T21:35:21.141626+0000","last_became_active":"2024-03-26T21:32:55.470413+0000","last_became_peered":"2024-03-26T21:32:55.470413+0000","last_unstale":"2024-03-26T21:35:21.141626+0000","last_undegraded":"2024-03-26T21:35:21.141626+0000","last_fullsized":"2024-03-26T21:35:21.141626+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:12.660555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.660555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.356026+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.144772+0000","last_change":"2024-03-26T21:34:26.592280+0000","last_active":"2024-03-26T21:35:21.144772+0000","last_peered":"2024-03-26T21:35:21.144772+0000","last_clean":"2024-03-26T21:35:21.144772+0000","last_became_active":"2024-03-26T21:32:55.472013+0000","last_became_peered":"2024-03-26T21:32:55.472013+0000","last_unstale":"2024-03-26T21:35:21.144772+0000","last_undegraded":"2024-03-26T21:35:21.144772+0000","last_fullsized":"2024-03-26T21:35:21.144772+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.591961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.591961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:28.616850+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":269,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.077443+0000","last_change":"2024-03-26T21:34:33.006206+0000","last_active":"2024-03-26T21:35:21.077443+0000","last_peered":"2024-03-26T21:35:21.077443+0000","last_clean":"2024-03-26T21:35:21.077443+0000","last_became_active":"2024-03-26T21:32:55.469205+0000","last_became_peered":"2024-03-26T21:32:55.469205+0000","last_unstale":"2024-03-26T21:35:21.077443+0000","last_undegraded":"2024-03-26T21:35:21.077443+0000","last_fullsized":"2024-03-26T21:35:21.077443+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.006101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.004115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":203,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.128966+0000","last_change":"2024-03-26T21:35:13.172081+0000","last_active":"2024-03-26T21:35:28.128966+0000","last_peered":"2024-03-26T21:35:28.128966+0000","last_clean":"2024-03-26T21:35:28.128966+0000","last_became_active":"2024-03-26T21:32:55.464394+0000","last_became_peered":"2024-03-26T21:32:55.464394+0000","last_unstale":"2024-03-26T21:35:28.128966+0000","last_undegraded":"2024-03-26T21:35:28.128966+0000","last_fullsized":"2024-03-26T21:35:28.128966+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:35:13.171864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.171864+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.108545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"178'2479","reported_seq":2499,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:26.989954+0000","last_change":"2024-03-26T21:34:58.692961+0000","last_active":"2024-03-26T21:35:26.989954+0000","last_peered":"2024-03-26T21:35:26.989954+0000","last_clean":"2024-03-26T21:35:26.989954+0000","last_became_active":"2024-03-26T21:34:58.692182+0000","last_became_peered":"2024-03-26T21:34:58.692182+0000","last_unstale":"2024-03-26T21:35:26.989954+0000","last_undegraded":"2024-03-26T21:35:26.989954+0000","last_fullsized":"2024-03-26T21:35:26.989954+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":2479,"log_dups_size":0,"ondisk_log_size":2479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.418350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162463744,"num_objects":2479,"num_object_clones":0,"num_object_copies":7437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":158656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":164,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.129329+0000","last_change":"2024-03-26T21:35:15.211309+0000","last_active":"2024-03-26T21:35:28.129329+0000","last_peered":"2024-03-26T21:35:28.129329+0000","last_clean":"2024-03-26T21:35:28.129329+0000","last_became_active":"2024-03-26T21:32:55.471347+0000","last_became_peered":"2024-03-26T21:32:55.471347+0000","last_unstale":"2024-03-26T21:35:28.129329+0000","last_undegraded":"2024-03-26T21:35:28.129329+0000","last_fullsized":"2024-03-26T21:35:28.129329+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.211195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.211195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.809000+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":343,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:24.970003+0000","last_change":"2024-03-26T21:34:19.761741+0000","last_active":"2024-03-26T21:35:24.970003+0000","last_peered":"2024-03-26T21:35:24.970003+0000","last_clean":"2024-03-26T21:35:24.970003+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:35:24.970003+0000","last_undegraded":"2024-03-26T21:35:24.970003+0000","last_fullsized":"2024-03-26T21:35:24.970003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:34:19.761382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:19.761382+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:30.563424+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-26T21:35:21.075050+0000","last_change":"2024-03-26T21:34:07.273064+0000","last_active":"2024-03-26T21:35:21.075050+0000","last_peered":"2024-03-26T21:35:21.075050+0000","last_clean":"2024-03-26T21:35:21.075050+0000","last_became_active":"2024-03-26T21:32:55.473354+0000","last_became_peered":"2024-03-26T21:32:55.473354+0000","last_unstale":"2024-03-26T21:35:21.075050+0000","last_undegraded":"2024-03-26T21:35:21.075050+0000","last_fullsized":"2024-03-26T21:35:21.075050+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.272948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.272948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:31.339263+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":186,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-26T21:35:28.128023+0000","last_change":"2024-03-26T21:34:14.786736+0000","last_active":"2024-03-26T21:35:28.128023+0000","last_peered":"2024-03-26T21:35:28.128023+0000","last_clean":"2024-03-26T21:35:28.128023+0000","last_became_active":"2024-03-26T21:32:55.476367+0000","last_became_peered":"2024-03-26T21:32:55.476367+0000","last_unstale":"2024-03-26T21:35:28.128023+0000","last_undegraded":"2024-03-26T21:35:28.128023+0000","last_fullsized":"2024-03-26T21:35:28.128023+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:34:14.786394+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.786394+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:29.339829+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2599288832,"num_objects":39662,"num_object_clones":0,"num_object_copies":118986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39662,"num_write_kb":2538368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7810056192,"data_stored":7810056192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39662,"ondisk_log_size":39662,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070551,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871072,"kb_used_data":753816,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92874080,"statfs":{"total":95995035648,"available":95103057920,"internally_reserved":0,"allocated":771907584,"data_stored":771481438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364893,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890752,"kb_used_data":773496,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92854400,"statfs":{"total":95995035648,"available":95082905600,"internally_reserved":0,"allocated":792059904,"data_stored":791664114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":120060358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252336,"kb_used_data":1098984,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92492816,"statfs":{"total":95995035648,"available":94712643584,"internally_reserved":0,"allocated":1125359616,"data_stored":1124915993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097632,"kb_used_data":926232,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92647520,"statfs":{"total":95995035648,"available":94871060480,"internally_reserved":0,"allocated":948461568,"data_stored":948034728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067448,"kb_used_data":923824,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92677704,"statfs":{"total":95995035648,"available":94901968896,"internally_reserved":0,"allocated":945995776,"data_stored":945607749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1204836,"kb_used_data":930716,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92540316,"statfs":{"total":95995035648,"available":94761283584,"internally_reserved":0,"allocated":953053184,"data_stored":952620890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650400,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006192,"kb_used_data":829668,"kb_used_omap":4,"kb_used_meta":176507,"kb_avail":92738960,"statfs":{"total":95995035648,"available":94964695040,"internally_reserved":0,"allocated":849580032,"data_stored":849147551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":180743843},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659232,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322448,"kb_used_data":1223240,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92422704,"statfs":{"total":95995035648,"available":94640848896,"internally_reserved":0,"allocated":1252597760,"data_stored":1252185726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":984547328,"data_stored":984547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1134559232,"data_stored":1134559232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":967376896,"data_stored":967376896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":985923584,"data_stored":985923584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":864223232,"data_stored":864223232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1277755392,"data_stored":1277755392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":801701888,"data_stored":801701888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":793968640,"data_stored":793968640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:35:31.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:31.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:31.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:31.922 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:31.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:31.982 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 23 16 48313 48297 131.213 100.75 0.00864608 0.00761631 2024-03-26T21:35:32.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:32.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:32.287 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:35:32.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:35:32.302 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:32 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:35:32.336 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:32.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:35:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:32 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:35:32.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:32.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:32.710 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:32.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:32.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:32.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:32.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:32.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 24 16 49896 49880 129.868 98.9375 0.00628889 0.00769499 2024-03-26T21:35:32.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:33.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:33.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:33.185 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:35:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:33.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:33.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:33.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:33.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:33 smithi067 ceph-mon[33346]: pgmap v288: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T21:35:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:33 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:35:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:33 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:35:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3307545621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:33 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[29406]: pgmap v288: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3307545621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[40587]: pgmap v288: 148 pgs: 148 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:35:33.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3307545621' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.526 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:33 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:35:33.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:33.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:33.746 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:33.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:33.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:33.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:33.980 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 25 16 51576 51560 128.873 105 0.00635809 0.00775386 2024-03-26T21:35:33.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:34.028 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:34.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:34.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:34.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:34.300 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:34.402 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.444 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:34.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:35:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:35:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:35:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:35:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:35:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:35:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:35:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:34 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:35:34.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:35:34.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:35:34.492 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:35:34.492 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:34 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:35:34.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:34.578 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:34.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:34.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:35:34.709 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:34.811 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:34.859 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:34.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:34.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:34.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 26 16 53144 53128 127.674 98 0.0112151 0.00782618 2024-03-26T21:35:35.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:35.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:35.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:35.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:35.356 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:35:35.357 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:35:35.358 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:35:35.358 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:35:35.358 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:35:35.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:35.471 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:35 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:35:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:35 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:35:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:35 smithi067 ceph-mon[33346]: pgmap v290: 148 pgs: 148 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-26T21:35:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4207174222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[40587]: pgmap v290: 148 pgs: 148 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4207174222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[29406]: pgmap v290: 148 pgs: 148 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-26T21:35:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4207174222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:35.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:35.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:35.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:35.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:35.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:35.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:35.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:35.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 27 16 54733 54717 126.623 99.3125 0.00643249 0.00789226 2024-03-26T21:35:35.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:36.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:36.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:36.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.300 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:36.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:36.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:36 smithi055 ceph-mon[29406]: pgmap v291: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T21:35:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:36 smithi055 ceph-mon[40587]: pgmap v291: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T21:35:36.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:36.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:36 smithi067 ceph-mon[33346]: pgmap v291: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-26T21:35:36.529 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:36.542 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.608 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:36.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:36.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:36.761 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:36.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:36.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:36.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:36.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 28 16 56288 56272 125.572 97.1875 0.0146277 0.00795836 2024-03-26T21:35:37.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:37.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:37.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:37.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:37.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:37.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:37.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:35:37.313 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:35:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:35:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:35:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:35:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:37.415 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:37.467 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T21:35:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:37.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:37.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:37.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:37.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:37.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:37.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:37.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:37.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 29 16 57794 57778 124.487 94.125 0.00985328 0.00802774 2024-03-26T21:35:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:38.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:38.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:35:38.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:38.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:38.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.307 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:38.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:38.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:35:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:35:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:35:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:35:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:35:38.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:35:38.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:38 smithi067 ceph-mon[33346]: pgmap v293: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-26T21:35:38.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:35:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:35:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:35:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[40587]: pgmap v293: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:35:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-mon[29406]: pgmap v293: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-26T21:35:38.570 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:38.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:38.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:38.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:38.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:38.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:38.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:38.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:38.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 30 16 58940 58924 122.725 71.625 0.00280186 0.00812737 2024-03-26T21:35:39.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:39.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:39.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:39.240 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:39.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:39.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.517 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:39.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:39.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:35:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:39 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:35:39.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:39.657 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:39.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:39.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:35:39.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:39 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:35:39.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:39.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:39.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:39.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:39.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 31 16 60298 60282 121.504 84.875 0.0237223 0.00822455 2024-03-26T21:35:40.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:40.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:40.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:40.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[29406]: pgmap v295: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 821 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[40587]: pgmap v295: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 821 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:35:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:40 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:35:40.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.351 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:40.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:40.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:40.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:40.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:40.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:40.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:40 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:35:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:40 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:35:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:40 smithi067 ceph-mon[33346]: pgmap v295: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 821 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T21:35:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:40 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:35:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:40 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:35:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:40.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:40.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:40.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:40.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 32 16 61700 61684 120.445 87.625 0.00364549 0.00829392 2024-03-26T21:35:41.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:41.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:41.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:41.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:41.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:41.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:41.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:41.447 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:35:41.447 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:35:41.447 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:35:41.448 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:35:41.448 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:35:41.448 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:35:41.448 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:35:41.448 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:41 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:35:41.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:41.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:41.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:41 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:35:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:41 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:35:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:41 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:35:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:41 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:35:41.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:41.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:41.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:41.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 33 16 62990 62974 119.239 80.625 0.0131557 0.00838168 2024-03-26T21:35:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:41.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:41.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:42.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:42.121 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:42.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:42.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:1121 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94208173342592 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:30:30.731602+0000", 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 310.67018565799998, 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0174519449999999, 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:35:42.277 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 1121 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731602+0000", 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731602+0000", 2024-03-26T21:35:42.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731604+0000", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.531e-06 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731610+0000", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6989999999999997e-06 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731626+0000", 2024-03-26T21:35:42.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5488000000000001e-05 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731637+0000", 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1088e-05 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731713+0000", 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6194999999999998e-05 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:35:42.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.731720+0000", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9260000000000003e-06 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.744904+0000", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.013183579000000001 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:30.744914+0000", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8449999999999998e-06 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:31.746563+0000", 2024-03-26T21:35:42.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.001649376 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:31.746870+0000", 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030673799999999998 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:30:31.749054+0000", 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0021844799999999999 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:118014 7.3 7:c6ed93ee:::benchmark_data_smithi055_81491_object118013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:32:21.936138+0000", 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 199.46564981, 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.138179894, 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 118014 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.283 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936138+0000", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936138+0000", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936139+0000", 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3600000000000004e-07 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936164+0000", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.5806e-05 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936173+0000", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4260000000000003e-06 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.936180+0000", 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.357e-06 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.941195+0000", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050152570000000004 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.941309+0000", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113509 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.941447+0000", 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013834200000000001 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.073878+0000", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13243065800000001 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.073921+0000", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2932999999999997e-05 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.074230+0000", 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030916299999999999 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.074318+0000", 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7906999999999997e-05 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:118029 7.2 7:42e4a756:::benchmark_data_smithi055_81491_object118028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:32:21.947524+0000", 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 199.454263731, 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15027122900000001, 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.288 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 118029 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.947524+0000", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.947524+0000", 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.947526+0000", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4079999999999999e-06 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:21.947546+0000", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0149e-05 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.066510+0000", 2024-03-26T21:35:42.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.118963957 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.066522+0000", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2486999999999999e-05 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.073982+0000", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0074597020000000003 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.083918+0000", 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0099363090000000008 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.084070+0000", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015191400000000001 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.092524+0000", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0084534629999999996 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.092777+0000", 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025395700000000002 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.097700+0000", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0049230189999999998 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.097752+0000", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1823999999999997e-05 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:32:22.097795+0000", 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3040000000000001e-05 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196352 7.3 7:ca0b4200:::benchmark_data_smithi055_81491_object76865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.241903+0000", 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.159884672, 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.153081201, 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196352 2024-03-26T21:35:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241903+0000", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241903+0000", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241904+0000", 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6199999999999998e-07 2024-03-26T21:35:42.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241907+0000", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2710000000000001e-06 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241915+0000", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8590000000000005e-06 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.241921+0000", 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4330000000000001e-06 2024-03-26T21:35:42.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242725+0000", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00080367000000000004 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.250172+0000", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0074471829999999996 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.250290+0000", 2024-03-26T21:35:42.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011772799999999999 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.257597+0000", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0073073460000000002 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388093+0000", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13049565299999999 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.394888+0000", 2024-03-26T21:35:42.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0067948310000000003 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.394939+0000", 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1236000000000002e-05 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.394984+0000", 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5429000000000002e-05 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196357 7.3 7:c2afc113:::benchmark_data_smithi055_81491_object76870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.242774+0000", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.159013738, 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.152514135, 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196357 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242774+0000", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242774+0000", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242775+0000", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4099999999999998e-07 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242779+0000", 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.014e-06 2024-03-26T21:35:42.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242786+0000", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.737e-06 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.242792+0000", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.071e-06 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.254985+0000", 2024-03-26T21:35:42.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012193176999999999 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.255090+0000", 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105145 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.255205+0000", 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011476000000000001 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.303 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.257642+0000", 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024372489999999998 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388098+0000", 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.130456497 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395199+0000", 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0071008979999999996 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395244+0000", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4913999999999997e-05 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395288+0000", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3931999999999999e-05 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196362 7.a 7:537e3031:::benchmark_data_smithi055_81491_object76875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.246235+0000", 2024-03-26T21:35:42.305 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.155553189, 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14606992299999999, 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196362 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246235+0000", 2024-03-26T21:35:42.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246235+0000", 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246236+0000", 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2109999999999999e-06 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246240+0000", 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2270000000000004e-06 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246247+0000", 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2649999999999999e-06 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.246253+0000", 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8289999999999996e-06 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.248395+0000", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0021418610000000001 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.248504+0000", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010907499999999999 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,5", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.248611+0000", 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010726 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.309 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.256672+0000", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0080601200000000005 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.256931+0000", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000259361 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392202+0000", 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.135271428 2024-03-26T21:35:42.310 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392258+0000", 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5656999999999999e-05 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392305+0000", 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6628999999999997e-05 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196363 7.3 7:cd8b422d:::benchmark_data_smithi055_81491_object76876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.311 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.247201+0000", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.15458658999999, 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.148195246, 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196363 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247201+0000", 2024-03-26T21:35:42.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247201+0000", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247202+0000", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5499999999999996e-07 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247207+0000", 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.011e-06 2024-03-26T21:35:42.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247218+0000", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0376e-05 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.247225+0000", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3119999999999996e-06 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.257672+0000", 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010446825 2024-03-26T21:35:42.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.257791+0000", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000119568 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.257897+0000", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010559 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388080+0000", 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13018276100000001 2024-03-26T21:35:42.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388104+0000", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.463e-05 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395323+0000", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072186359999999996 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395363+0000", 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0268000000000001e-05 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.316 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395396+0000", 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3314e-05 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196364 7.a 7:503994e7:::benchmark_data_smithi055_81491_object76877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.249845+0000", 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.151942397, 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14258149000000001, 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.317 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196364 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249845+0000", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.318 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249845+0000", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249846+0000", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0139999999999999e-06 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249852+0000", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4650000000000002e-06 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249861+0000", 2024-03-26T21:35:42.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0149999999999992e-06 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.249869+0000", 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.627e-06 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253652+0000", 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037830759999999998 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.320 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253756+0000", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104202 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,5", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253860+0000", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010392599999999999 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.256851+0000", 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029917279999999999 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.321 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.256937+0000", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5257e-05 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392345+0000", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13540819200000001 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392390+0000", 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4965000000000001e-05 2024-03-26T21:35:42.322 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.392427+0000", 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7023000000000001e-05 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:196372 7.3 7:c5f6e2ee:::benchmark_data_smithi055_81491_object76885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:47.253625+0000", 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 114.148163129, 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14187275099999999, 2024-03-26T21:35:42.323 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 196372 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253625+0000", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.324 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253625+0000", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253626+0000", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4099999999999997e-07 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253630+0000", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3309999999999997e-06 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.325 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253638+0000", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2379999999999997e-06 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.253646+0000", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.435e-06 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.260211+0000", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.006565825 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.326 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.260318+0000", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000106455 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.260425+0000", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000106718 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388109+0000", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.12768491500000001 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.327 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.388152+0000", 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2583000000000002e-05 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395428+0000", 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072755809999999997 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395466+0000", 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8683999999999998e-05 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.328 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:47.395497+0000", 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1145000000000001e-05 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197261 7.3 7:ca90de6b:::benchmark_data_smithi055_81491_object77774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.397538+0000", 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 113.004250051, 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.146085875, 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.329 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197261 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397538+0000", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.330 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397538+0000", 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397538+0000", 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4700000000000002e-07 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397541+0000", 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7480000000000002e-06 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.331 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397547+0000", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7579999999999998e-06 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.397552+0000", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4620000000000004e-06 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.402371+0000", 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0048183669999999996 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.332 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.402479+0000", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000108763 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.402588+0000", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109076 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.407841+0000", 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0052522649999999999 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.333 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.407890+0000", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9666000000000002e-05 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543525+0000", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13563441900000001 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543579+0000", 2024-03-26T21:35:42.334 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4527999999999997e-05 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543624+0000", 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4376000000000003e-05 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197270 7.a 7:50a441d1:::benchmark_data_smithi055_81491_object77783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.405838+0000", 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.995949822, 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14340465299999999, 2024-03-26T21:35:42.335 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197270 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405838+0000", 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.336 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405838+0000", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405839+0000", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.141e-06 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405844+0000", 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4469999999999999e-06 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.337 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405854+0000", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0293e-05 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405861+0000", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5769999999999996e-06 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.405983+0000", 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121107 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.338 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406241+0000", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025803499999999999 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,5", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406347+0000", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010687399999999999 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.413713+0000", 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0073656490000000002 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.416371+0000", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0026581539999999998 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549095+0000", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13272402599999999 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549193+0000", 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7955000000000007e-05 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.340 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549243+0000", 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9394999999999997e-05 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197271 7.2 7:4770ee7c:::benchmark_data_smithi055_81491_object77784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.406029+0000", 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.995758733, 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.143370161, 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.341 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197271 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406029+0000", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406029+0000", 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406030+0000", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3600000000000003e-07 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406034+0000", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.515e-06 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406042+0000", 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6669999999999996e-06 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.406049+0000", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6719999999999998e-06 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.407544+0000", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0014957729999999999 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.407655+0000", 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000110523 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.344 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.407766+0000", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111353 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.416413+0000", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0086469630000000006 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549027+0000", 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13261377399999999 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.345 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549291+0000", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000263706 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549341+0000", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0049999999999997e-05 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549399+0000", 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8428999999999999e-05 2024-03-26T21:35:42.346 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197274 7.2 7:4a88e402:::benchmark_data_smithi055_81491_object77787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.410481+0000", 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.991307171, 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.139007407, 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197274 2024-03-26T21:35:42.347 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410481+0000", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410481+0000", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410482+0000", 2024-03-26T21:35:42.348 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.415e-06 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410488+0000", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3260000000000001e-06 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410501+0000", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2812e-05 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.410510+0000", 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.653e-06 2024-03-26T21:35:42.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.416449+0000", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0059395059999999998 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.416561+0000", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111407 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.416671+0000", 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011018000000000001 2024-03-26T21:35:42.350 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549077+0000", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13240626899999999 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549184+0000", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000106638 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549408+0000", 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022388599999999999 2024-03-26T21:35:42.351 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549453+0000", 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5646000000000003e-05 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549488+0000", 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4669000000000003e-05 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197275 7.3 7:c247b85b:::benchmark_data_smithi055_81491_object77788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.352 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.411018+0000", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.990770189, 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13289434, 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197275 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411018+0000", 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.353 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411018+0000", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411018+0000", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4200000000000005e-07 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411023+0000", 2024-03-26T21:35:42.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7169999999999999e-06 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411032+0000", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6840000000000002e-06 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411039+0000", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9500000000000004e-06 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411100+0000", 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1446999999999996e-05 2024-03-26T21:35:42.355 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411214+0000", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113712 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411326+0000", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112353 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414812+0000", 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034861229999999998 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543639+0000", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.128826249 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543812+0000", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000173635 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543859+0000", 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6397999999999999e-05 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543912+0000", 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3229999999999997e-05 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197276 7.3 7:cf25280f:::benchmark_data_smithi055_81491_object77789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.411902+0000", 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.989885786, 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13215471400000001, 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.358 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197276 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411902+0000", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.359 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411902+0000", 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411903+0000", 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2900000000000002e-07 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411908+0000", 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9309999999999999e-06 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411918+0000", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7399999999999999e-06 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.411925+0000", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3839999999999999e-06 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414843+0000", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029183059999999999 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414946+0000", 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000102477 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,4", 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.415050+0000", 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010399999999999999 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543645+0000", 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.12859496100000001 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.362 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543714+0000", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9645000000000002e-05 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.543954+0000", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023942399999999999 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.544009+0000", 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5714000000000002e-05 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.363 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.544057+0000", 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7302999999999998e-05 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197278 7.2 7:4042f8bd:::benchmark_data_smithi055_81491_object77791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.414118+0000", 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.987669938, 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13544319099999999, 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.364 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197278 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414118+0000", 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.365 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414118+0000", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414119+0000", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.562e-06 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414124+0000", 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1450000000000001e-06 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.366 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414133+0000", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0259999999999996e-06 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.414139+0000", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.579e-06 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.419208+0000", 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050685139999999997 2024-03-26T21:35:42.367 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.419311+0000", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010305400000000001 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.419416+0000", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104832 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549225+0000", 2024-03-26T21:35:42.368 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.12980899900000001 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549261+0000", 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6007000000000003e-05 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549497+0000", 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023613400000000001 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.369 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549531+0000", 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4334e-05 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.549561+0000", 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0005000000000002e-05 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197499 7.1 7:8077d86a:::benchmark_data_smithi055_81491_object78012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.738951+0000", 2024-03-26T21:35:42.370 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.662836724, 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14691666, 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197499 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738951+0000", 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.371 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738951+0000", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738952+0000", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1930000000000001e-06 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738957+0000", 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5360000000000003e-06 2024-03-26T21:35:42.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738967+0000", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0569e-05 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.738976+0000", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3329999999999997e-06 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.741012+0000", 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002036052 2024-03-26T21:35:42.373 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.741143+0000", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013130499999999999 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 4,6", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.741282+0000", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013936600000000001 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.743285+0000", 2024-03-26T21:35:42.374 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020029869999999999 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.743471+0000", 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000185248 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.743487+0000", 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6668999999999999e-05 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.743531+0000", 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.392e-05 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.885868+0000", 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14233648199999999 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197502 7.2 7:434673d2:::benchmark_data_smithi055_81491_object78015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.742654+0000", 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.65913383900001, 2024-03-26T21:35:42.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14793632600000001, 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197502 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742654+0000", 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.377 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742654+0000", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742655+0000", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1289999999999999e-06 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742661+0000", 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7649999999999999e-06 2024-03-26T21:35:42.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742672+0000", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0978999999999999e-05 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742688+0000", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6035000000000001e-05 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742726+0000", 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8454999999999997e-05 2024-03-26T21:35:42.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742799+0000", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2229000000000003e-05 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.742864+0000", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5031999999999997e-05 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886091+0000", 2024-03-26T21:35:42.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14322784599999999 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886275+0000", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018325700000000001 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.890486+0000", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0042111429999999997 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.890546+0000", 2024-03-26T21:35:42.381 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9783e-05 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.890590+0000", 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4672999999999997e-05 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.24704.0:197503 7.7 e151/95 7:e387ae8a:::benchmark_data_smithi055_81491_object78016:head v 151'12573, mlcod=151'12571)", 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.744910+0000", 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.656878083, 2024-03-26T21:35:42.382 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.150814806, 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.744910+0000", 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.744910+0000", 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.383 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.744911+0000", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1200000000000002e-07 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.744917+0000", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1659999999999999e-06 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886135+0000", 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.141218494 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.384 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886143+0000", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.191e-06 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886568+0000", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00042412999999999999 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886592+0000", 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4346999999999999e-05 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.385 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.895649+0000", 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0090567189999999995 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.895725+0000", 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5946999999999996e-05 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.24704.0:197502 7.2 e151/95 ondisk, result = 0)", 2024-03-26T21:35:42.386 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:48.745260+0000", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 112.65652781199999, 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.140840724, 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.745260+0000", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.745260+0000", 2024-03-26T21:35:42.387 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.745261+0000", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9400000000000005e-07 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.745266+0000", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6049999999999997e-06 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.885910+0000", 2024-03-26T21:35:42.388 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14064400699999999 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.885921+0000", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1e-05 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886075+0000", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000154117 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886084+0000", 2024-03-26T21:35:42.389 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8959999999999992e-06 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:48.886101+0000", 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6405e-05 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:35:42.390 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:35:42.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:42.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:42.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:42.785 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:42.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:42.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:42.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:42.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 34 16 64450 64434 118.415 91.25 0.00748654 0.00843972 2024-03-26T21:35:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:42 smithi067 ceph-mon[33346]: pgmap v296: 148 pgs: 148 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T21:35:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:42 smithi067 ceph-mon[33346]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:35:43.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:43.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:42 smithi055 ceph-mon[40587]: pgmap v296: 148 pgs: 148 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T21:35:43.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:42 smithi055 ceph-mon[40587]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:35:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:42 smithi055 ceph-mon[29406]: pgmap v296: 148 pgs: 148 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T21:35:43.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:42 smithi055 ceph-mon[29406]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T21:35:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:43.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:43.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:43.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:43.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:43.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:43.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:43.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:43.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:43.760 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:43.861 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:43.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:35:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:43.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:43.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:43.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:43 smithi067 ceph-mon[33346]: 5.11 deep-scrub starts 2024-03-26T21:35:43.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:43 smithi067 ceph-mon[33346]: 5.11 deep-scrub ok 2024-03-26T21:35:43.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 35 16 65888 65872 117.6 89.875 0.00969232 0.00849871 2024-03-26T21:35:43.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:43 smithi055 ceph-mon[29406]: 5.11 deep-scrub starts 2024-03-26T21:35:43.999 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:43 smithi055 ceph-mon[29406]: 5.11 deep-scrub ok 2024-03-26T21:35:44.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:43 smithi055 ceph-mon[40587]: 5.11 deep-scrub starts 2024-03-26T21:35:44.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:43 smithi055 ceph-mon[40587]: 5.11 deep-scrub ok 2024-03-26T21:35:44.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:44.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:44.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:44.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:44.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:35:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:44.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:44.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:44.502 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:44.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:44.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:44.697 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:44 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:35:44.697 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:44 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:35:44.697 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:44 smithi067 ceph-mon[33346]: pgmap v298: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 755 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T21:35:44.710 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:35:44.710 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:35:44.711 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[29406]: pgmap v298: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 755 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T21:35:44.711 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:35:44.711 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:35:44.711 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:44 smithi055 ceph-mon[40587]: pgmap v298: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 755 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T21:35:44.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:44.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:44.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:44.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:44.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 36 16 67280 67264 116.75 87 0.0238441 0.0085603 2024-03-26T21:35:45.013 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:45.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:45.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:45.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:45.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:45.345 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:45.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:45.492 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:45.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:45.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:45.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:45.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:45.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:45.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:45.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 37 16 68623 68607 115.863 83.9375 0.0136596 0.00862546 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:45 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:35:45.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:46.026 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:35:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:35:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:35:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:35:46.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:35:46.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:35:46.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:35:46.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:35:46.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:35:46.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:35:46.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:35:46.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:45 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:35:46.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:46.162 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:46.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:46.356 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:35:46.361 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:35:46.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:46.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.590 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:46.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:46.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:46.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:46.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:46.926 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:35:46.926 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:35:46.926 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[40587]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:46.927 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:46 smithi055 ceph-mon[29406]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:35:46.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:35:46.959 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:46 smithi067 ceph-mon[33346]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T21:35:46.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:46.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 38 16 69901 69885 114.916 79.875 0.00860321 0.00869614 2024-03-26T21:35:47.005 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:47.048 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:35:47.048 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:35:47.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:69967 8.0 e184/173 8:092b0bf7:::benchmark_data_smithi055_110126_object69966:head v 184'4280, mlcod=184'4277)", 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:47.049571+0000", 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0041835960000000004, 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042192469999999998, 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:47.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049571+0000", 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049571+0000", 2024-03-26T21:35:47.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049571+0000", 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8000000000000005e-07 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049632+0000", 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0174999999999999e-05 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049646+0000", 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4384e-05 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049653+0000", 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1430000000000001e-06 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.050590+0000", 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00093672000000000004 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.050609+0000", 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9091999999999999e-05 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:69969 8.8 8:170da62b:::benchmark_data_smithi055_110126_object69968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:47.049766+0000", 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0039882759999999998, 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041064099999999996, 2024-03-26T21:35:47.054 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 69969 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049766+0000", 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049766+0000", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049767+0000", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3209999999999999e-06 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049803+0000", 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5596000000000003e-05 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049813+0000", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0030999999999999e-05 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.049821+0000", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7819999999999994e-06 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.051239+0000", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001418575 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:35:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.051375+0000", 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013560300000000001 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.051518+0000", 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000143478 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:69968 8.a e184/173 8:52c17801:::benchmark_data_smithi055_110126_object69967:head v 184'4323, mlcod=184'4320)", 2024-03-26T21:35:47.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:47.052753+0000", 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0010011359999999999, 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001206384, 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:47.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052753+0000", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052753+0000", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052754+0000", 2024-03-26T21:35:47.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7600000000000006e-07 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052784+0000", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9632e-05 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052794+0000", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349999999999999e-05 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052802+0000", 2024-03-26T21:35:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0560000000000005e-06 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052876+0000", 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4340999999999999e-05 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:47.052897+0000", 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0517e-05 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:35:47.063 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 3 2024-03-26T21:35:47.063 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:35:47.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:47.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:47.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:47.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:47.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:47.805 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:47.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:35:47.840 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:47.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:47.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 39 16 70959 70943 113.665 66.125 0.0147412 0.00878935 2024-03-26T21:35:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:47.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:47.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:47 smithi067 ceph-mon[33346]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:47.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:47.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[29406]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:48.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[40587]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:35:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:35:48.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:48.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:48.207 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:35:48.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:48.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:48.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:48.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:48.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:48.542 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:48.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:48.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:48.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:48.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: pgmap v301: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: pgmap v301: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T21:35:48.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:35:48.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:35:48.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:35:48.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:35:48.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:35:48.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:48 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:35:48.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:48.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: pgmap v301: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:35:48.862 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:35:48.863 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:48 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:35:48.907 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:35:48.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:48.955 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:35:48.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:35:48.992472+0000 min lat: 0.0012867 max lat: 0.203486 avg lat: 0.00887093 2024-03-26T21:35:48.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:35:48.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 40 16 72123 72107 112.642 72.75 0.0210769 0.00887093 2024-03-26T21:35:48.993 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:35:49.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:49.039 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:35:49.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:35:49.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:49.136 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:35:49.179 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:49.179 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:35:49.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:49.221 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:35:49.230 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:35:49.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:49.338 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:49.440 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:49.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:49.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:49.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:49.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:49.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:49.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:49.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:49.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/503345996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 3.1d deep-scrub starts 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 3.1d deep-scrub ok 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:49.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/503345996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 3.1d deep-scrub starts 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 3.1d deep-scrub ok 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:49.951 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:49.952 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:35:49.952 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:49 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:35:49.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:49.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 41 16 73221 73205 111.568 68.625 0.00756597 0.00895819 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/503345996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 3.1d deep-scrub starts 2024-03-26T21:35:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 3.1d deep-scrub ok 2024-03-26T21:35:49.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:35:49.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:35:49.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:35:49.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:35:49.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:49 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:35:50.022 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:50.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:50.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:50.254 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-03-26T21:35:45.416198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4112062888,"num_objects":62934,"num_object_clones":0,"num_object_copies":188802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62934,"num_whiteouts":0,"num_read":1346,"num_read_kb":1192,"num_write":63351,"num_write_kb":4017565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63403,"ondisk_log_size":63403,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13466784,"kb_used_data":11973908,"kb_used_omap":15,"kb_used_meta":1492784,"kb_avail":736494432,"statfs":{"total":767960285184,"available":754170298368,"internally_reserved":0,"allocated":12261281792,"data_stored":12257746434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1528611598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,10,5,1,4,0,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":475267072,"num_objects":7252,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7252,"num_write_kb":464128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004870"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":172,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636425+0000","last_change":"2024-03-26T21:35:18.215494+0000","last_active":"2024-03-26T21:35:41.636425+0000","last_peered":"2024-03-26T21:35:41.636425+0000","last_clean":"2024-03-26T21:35:41.636425+0000","last_became_active":"2024-03-26T21:32:55.466152+0000","last_became_peered":"2024-03-26T21:32:55.466152+0000","last_unstale":"2024-03-26T21:35:41.636425+0000","last_undegraded":"2024-03-26T21:35:41.636425+0000","last_fullsized":"2024-03-26T21:35:41.636425+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:18.215367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:18.215367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:47.706122+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266666+0000","last_change":"2024-03-26T21:34:20.509053+0000","last_active":"2024-03-26T21:35:38.266666+0000","last_peered":"2024-03-26T21:35:38.266666+0000","last_clean":"2024-03-26T21:35:38.266666+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:35:38.266666+0000","last_undegraded":"2024-03-26T21:35:38.266666+0000","last_fullsized":"2024-03-26T21:35:38.266666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.508804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.508804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637707+0000","last_change":"2024-03-26T21:35:27.960032+0000","last_active":"2024-03-26T21:35:41.637707+0000","last_peered":"2024-03-26T21:35:41.637707+0000","last_clean":"2024-03-26T21:35:41.637707+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:35:41.637707+0000","last_undegraded":"2024-03-26T21:35:41.637707+0000","last_fullsized":"2024-03-26T21:35:41.637707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.959913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.959913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:53.391417+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":200,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636160+0000","last_change":"2024-03-26T21:35:39.174442+0000","last_active":"2024-03-26T21:35:41.636160+0000","last_peered":"2024-03-26T21:35:41.636160+0000","last_clean":"2024-03-26T21:35:41.636160+0000","last_became_active":"2024-03-26T21:32:55.468198+0000","last_became_peered":"2024-03-26T21:32:55.468198+0000","last_unstale":"2024-03-26T21:35:41.636160+0000","last_undegraded":"2024-03-26T21:35:41.636160+0000","last_fullsized":"2024-03-26T21:35:41.636160+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:35:39.174314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:39.174314+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:54.329678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":257,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636590+0000","last_change":"2024-03-26T21:34:24.368408+0000","last_active":"2024-03-26T21:35:41.636590+0000","last_peered":"2024-03-26T21:35:41.636590+0000","last_clean":"2024-03-26T21:35:41.636590+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:35:41.636590+0000","last_undegraded":"2024-03-26T21:35:41.636590+0000","last_fullsized":"2024-03-26T21:35:41.636590+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.368161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.690462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":172,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636064+0000","last_change":"2024-03-26T21:34:27.847069+0000","last_active":"2024-03-26T21:35:41.636064+0000","last_peered":"2024-03-26T21:35:41.636064+0000","last_clean":"2024-03-26T21:35:41.636064+0000","last_became_active":"2024-03-26T21:32:55.468736+0000","last_became_peered":"2024-03-26T21:32:55.468736+0000","last_unstale":"2024-03-26T21:35:41.636064+0000","last_undegraded":"2024-03-26T21:35:41.636064+0000","last_fullsized":"2024-03-26T21:35:41.636064+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.846942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.846942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.712678+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637207+0000","last_change":"2024-03-26T21:34:30.380605+0000","last_active":"2024-03-26T21:35:41.637207+0000","last_peered":"2024-03-26T21:35:41.637207+0000","last_clean":"2024-03-26T21:35:41.637207+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:35:41.637207+0000","last_undegraded":"2024-03-26T21:35:41.637207+0000","last_fullsized":"2024-03-26T21:35:41.637207+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:30.379661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.379661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":290,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636371+0000","last_change":"2024-03-26T21:35:24.162334+0000","last_active":"2024-03-26T21:35:41.636371+0000","last_peered":"2024-03-26T21:35:41.636371+0000","last_clean":"2024-03-26T21:35:41.636371+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:35:41.636371+0000","last_undegraded":"2024-03-26T21:35:41.636371+0000","last_fullsized":"2024-03-26T21:35:41.636371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:35:24.162191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:24.162191+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.771496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":257,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635100+0000","last_change":"2024-03-26T21:35:25.946247+0000","last_active":"2024-03-26T21:35:41.635100+0000","last_peered":"2024-03-26T21:35:41.635100+0000","last_clean":"2024-03-26T21:35:41.635100+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:35:41.635100+0000","last_undegraded":"2024-03-26T21:35:41.635100+0000","last_fullsized":"2024-03-26T21:35:41.635100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.946125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.946125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.103970+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252829+0000","last_change":"2024-03-26T21:34:31.792345+0000","last_active":"2024-03-26T21:35:38.252829+0000","last_peered":"2024-03-26T21:35:38.252829+0000","last_clean":"2024-03-26T21:35:38.252829+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:35:38.252829+0000","last_undegraded":"2024-03-26T21:35:38.252829+0000","last_fullsized":"2024-03-26T21:35:38.252829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:31.792268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.792268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.823508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253021+0000","last_change":"2024-03-26T21:33:22.738629+0000","last_active":"2024-03-26T21:35:38.253021+0000","last_peered":"2024-03-26T21:35:38.253021+0000","last_clean":"2024-03-26T21:35:38.253021+0000","last_became_active":"2024-03-26T21:32:55.465246+0000","last_became_peered":"2024-03-26T21:32:55.465246+0000","last_unstale":"2024-03-26T21:35:38.253021+0000","last_undegraded":"2024-03-26T21:35:38.253021+0000","last_fullsized":"2024-03-26T21:35:38.253021+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:22.738510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:22.738510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"167'13","reported_seq":196,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636337+0000","last_change":"2024-03-26T21:34:30.923713+0000","last_active":"2024-03-26T21:35:41.636337+0000","last_peered":"2024-03-26T21:35:41.636337+0000","last_clean":"2024-03-26T21:35:41.636337+0000","last_became_active":"2024-03-26T21:32:55.480936+0000","last_became_peered":"2024-03-26T21:32:55.480936+0000","last_unstale":"2024-03-26T21:35:41.636337+0000","last_undegraded":"2024-03-26T21:35:41.636337+0000","last_fullsized":"2024-03-26T21:35:41.636337+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.923562+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252587+0000","last_change":"2024-03-26T21:34:22.762001+0000","last_active":"2024-03-26T21:35:38.252587+0000","last_peered":"2024-03-26T21:35:38.252587+0000","last_clean":"2024-03-26T21:35:38.252587+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:35:38.252587+0000","last_undegraded":"2024-03-26T21:35:38.252587+0000","last_fullsized":"2024-03-26T21:35:38.252587+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:22.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:34:22.761871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":148,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253498+0000","last_change":"2024-03-26T21:34:23.724557+0000","last_active":"2024-03-26T21:35:38.253498+0000","last_peered":"2024-03-26T21:35:38.253498+0000","last_clean":"2024-03-26T21:35:38.253498+0000","last_became_active":"2024-03-26T21:32:20.904068+0000","last_became_peered":"2024-03-26T21:32:20.904068+0000","last_unstale":"2024-03-26T21:35:38.253498+0000","last_undegraded":"2024-03-26T21:35:38.253498+0000","last_fullsized":"2024-03-26T21:35:38.253498+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.724419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.724419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:39.913429+0000","last_change":"2024-03-26T21:35:39.913471+0000","last_active":"2024-03-26T21:35:39.913429+0000","last_peered":"2024-03-26T21:35:39.913429+0000","last_clean":"2024-03-26T21:35:39.913429+0000","last_became_active":"2024-03-26T21:32:55.466491+0000","last_became_peered":"2024-03-26T21:32:55.466491+0000","last_unstale":"2024-03-26T21:35:39.913429+0000","last_undegraded":"2024-03-26T21:35:39.913429+0000","last_fullsized":"2024-03-26T21:35:39.913429+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:39.913327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:39.913327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:04.731040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":159,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636837+0000","last_change":"2024-03-26T21:35:36.208964+0000","last_active":"2024-03-26T21:35:41.636837+0000","last_peered":"2024-03-26T21:35:41.636837+0000","last_clean":"2024-03-26T21:35:41.636837+0000","last_became_active":"2024-03-26T21:32:55.476272+0000","last_became_peered":"2024-03-26T21:32:55.476272+0000","last_unstale":"2024-03-26T21:35:41.636837+0000","last_undegraded":"2024-03-26T21:35:41.636837+0000","last_fullsized":"2024-03-26T21:35:41.636837+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:35:36.208704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:36.208704+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:52.193376+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":137,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637550+0000","last_change":"2024-03-26T21:35:38.197519+0000","last_active":"2024-03-26T21:35:41.637550+0000","last_peered":"2024-03-26T21:35:41.637550+0000","last_clean":"2024-03-26T21:35:41.637550+0000","last_became_active":"2024-03-26T21:32:55.467335+0000","last_became_peered":"2024-03-26T21:32:55.467335+0000","last_unstale":"2024-03-26T21:35:41.637550+0000","last_undegraded":"2024-03-26T21:35:41.637550+0000","last_fullsized":"2024-03-26T21:35:41.637550+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:38.197384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.197384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:58.831160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267888+0000","last_change":"2024-03-26T21:35:16.465328+0000","last_active":"2024-03-26T21:35:38.267888+0000","last_peered":"2024-03-26T21:35:38.267888+0000","last_clean":"2024-03-26T21:35:38.267888+0000","last_became_active":"2024-03-26T21:32:55.472908+0000","last_became_peered":"2024-03-26T21:32:55.472908+0000","last_unstale":"2024-03-26T21:35:38.267888+0000","last_undegraded":"2024-03-26T21:35:38.267888+0000","last_fullsized":"2024-03-26T21:35:38.267888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:16.465203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:16.465203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:22.841819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638571+0000","last_change":"2024-03-26T21:34:27.088958+0000","last_active":"2024-03-26T21:35:41.638571+0000","last_peered":"2024-03-26T21:35:41.638571+0000","last_clean":"2024-03-26T21:35:41.638571+0000","last_became_active":"2024-03-26T21:32:55.479242+0000","last_became_peered":"2024-03-26T21:32:55.479242+0000","last_unstale":"2024-03-26T21:35:41.638571+0000","last_undegraded":"2024-03-26T21:35:41.638571+0000","last_fullsized":"2024-03-26T21:35:41.638571+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:27.088840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.088840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:49.279253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":298,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636220+0000","last_change":"2024-03-26T21:35:13.081542+0000","last_active":"2024-03-26T21:35:41.636220+0000","last_peered":"2024-03-26T21:35:41.636220+0000","last_clean":"2024-03-26T21:35:41.636220+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:35:41.636220+0000","last_undegraded":"2024-03-26T21:35:41.636220+0000","last_fullsized":"2024-03-26T21:35:41.636220+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:35:13.081138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.081138+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:22.799380+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.645910+0000","last_change":"2024-03-26T21:35:27.311601+0000","last_active":"2024-03-26T21:35:41.645910+0000","last_peered":"2024-03-26T21:35:41.645910+0000","last_clean":"2024-03-26T21:35:41.645910+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:35:41.645910+0000","last_undegraded":"2024-03-26T21:35:41.645910+0000","last_fullsized":"2024-03-26T21:35:41.645910+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.311473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.311473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:56.435056+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":172,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637392+0000","last_change":"2024-03-26T21:34:25.862079+0000","last_active":"2024-03-26T21:35:41.637392+0000","last_peered":"2024-03-26T21:35:41.637392+0000","last_clean":"2024-03-26T21:35:41.637392+0000","last_became_active":"2024-03-26T21:32:55.465947+0000","last_became_peered":"2024-03-26T21:32:55.465947+0000","last_unstale":"2024-03-26T21:35:41.637392+0000","last_undegraded":"2024-03-26T21:35:41.637392+0000","last_fullsized":"2024-03-26T21:35:41.637392+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.861995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.861995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.431656+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":184,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636774+0000","last_change":"2024-03-26T21:35:25.096014+0000","last_active":"2024-03-26T21:35:41.636774+0000","last_peered":"2024-03-26T21:35:41.636774+0000","last_clean":"2024-03-26T21:35:41.636774+0000","last_became_active":"2024-03-26T21:32:55.477250+0000","last_became_peered":"2024-03-26T21:32:55.477250+0000","last_unstale":"2024-03-26T21:35:41.636774+0000","last_undegraded":"2024-03-26T21:35:41.636774+0000","last_fullsized":"2024-03-26T21:35:41.636774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.095921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.095921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.118555+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":281,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636918+0000","last_change":"2024-03-26T21:34:09.286797+0000","last_active":"2024-03-26T21:35:41.636918+0000","last_peered":"2024-03-26T21:35:41.636918+0000","last_clean":"2024-03-26T21:35:41.636918+0000","last_became_active":"2024-03-26T21:32:55.470311+0000","last_became_peered":"2024-03-26T21:32:55.470311+0000","last_unstale":"2024-03-26T21:35:41.636918+0000","last_undegraded":"2024-03-26T21:35:41.636918+0000","last_fullsized":"2024-03-26T21:35:41.636918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:09.286636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.286636+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257734+0000","last_change":"2024-03-26T21:34:25.733995+0000","last_active":"2024-03-26T21:35:38.257734+0000","last_peered":"2024-03-26T21:35:38.257734+0000","last_clean":"2024-03-26T21:35:38.257734+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:35:38.257734+0000","last_undegraded":"2024-03-26T21:35:38.257734+0000","last_fullsized":"2024-03-26T21:35:38.257734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:34:25.733888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.733888+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"134'10","reported_seq":368,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259588+0000","last_change":"2024-03-26T21:34:27.049262+0000","last_active":"2024-03-26T21:35:38.259588+0000","last_peered":"2024-03-26T21:35:38.259588+0000","last_clean":"2024-03-26T21:35:38.259588+0000","last_became_active":"2024-03-26T21:32:55.477677+0000","last_became_peered":"2024-03-26T21:32:55.477677+0000","last_unstale":"2024-03-26T21:35:38.259588+0000","last_undegraded":"2024-03-26T21:35:38.259588+0000","last_fullsized":"2024-03-26T21:35:38.259588+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:27.048989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.048989+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:46.859023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":278,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266690+0000","last_change":"2024-03-26T21:35:12.512389+0000","last_active":"2024-03-26T21:35:38.266690+0000","last_peered":"2024-03-26T21:35:38.266690+0000","last_clean":"2024-03-26T21:35:38.266690+0000","last_became_active":"2024-03-26T21:32:55.469277+0000","last_became_peered":"2024-03-26T21:32:55.469277+0000","last_unstale":"2024-03-26T21:35:38.266690+0000","last_undegraded":"2024-03-26T21:35:38.266690+0000","last_fullsized":"2024-03-26T21:35:38.266690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:35:12.511752+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.511752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:16.863479+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":273,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637801+0000","last_change":"2024-03-26T21:32:22.112186+0000","last_active":"2024-03-26T21:35:41.637801+0000","last_peered":"2024-03-26T21:35:41.637801+0000","last_clean":"2024-03-26T21:35:41.637801+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:35:41.637801+0000","last_undegraded":"2024-03-26T21:35:41.637801+0000","last_fullsized":"2024-03-26T21:35:41.637801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:32:22.112001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:22.112001+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267726+0000","last_change":"2024-03-26T21:35:08.399748+0000","last_active":"2024-03-26T21:35:38.267726+0000","last_peered":"2024-03-26T21:35:38.267726+0000","last_clean":"2024-03-26T21:35:38.267726+0000","last_became_active":"2024-03-26T21:32:55.465456+0000","last_became_peered":"2024-03-26T21:32:55.465456+0000","last_unstale":"2024-03-26T21:35:38.267726+0000","last_undegraded":"2024-03-26T21:35:38.267726+0000","last_fullsized":"2024-03-26T21:35:38.267726+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:08.399643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.399643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:25.430769+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":262,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638062+0000","last_change":"2024-03-26T21:35:10.003783+0000","last_active":"2024-03-26T21:35:41.638062+0000","last_peered":"2024-03-26T21:35:41.638062+0000","last_clean":"2024-03-26T21:35:41.638062+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:35:41.638062+0000","last_undegraded":"2024-03-26T21:35:41.638062+0000","last_fullsized":"2024-03-26T21:35:41.638062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:10.003660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:35:10.003660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:27.965246+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":269,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635873+0000","last_change":"2024-03-26T21:35:20.257839+0000","last_active":"2024-03-26T21:35:41.635873+0000","last_peered":"2024-03-26T21:35:41.635873+0000","last_clean":"2024-03-26T21:35:41.635873+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:35:41.635873+0000","last_undegraded":"2024-03-26T21:35:41.635873+0000","last_fullsized":"2024-03-26T21:35:41.635873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:20.257696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:20.257696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:43.449764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"167'16","reported_seq":304,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259545+0000","last_change":"2024-03-26T21:35:29.097610+0000","last_active":"2024-03-26T21:35:38.259545+0000","last_peered":"2024-03-26T21:35:38.259545+0000","last_clean":"2024-03-26T21:35:38.259545+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:35:38.259545+0000","last_undegraded":"2024-03-26T21:35:38.259545+0000","last_fullsized":"2024-03-26T21:35:38.259545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'16","last_scrub_stamp":"2024-03-26T21:35:29.097470+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:35:29.097470+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:46.882258+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256560+0000","last_change":"2024-03-26T21:35:37.291623+0000","last_active":"2024-03-26T21:35:38.256560+0000","last_peered":"2024-03-26T21:35:38.256560+0000","last_clean":"2024-03-26T21:35:38.256560+0000","last_became_active":"2024-03-26T21:32:55.471141+0000","last_became_peered":"2024-03-26T21:32:55.471141+0000","last_unstale":"2024-03-26T21:35:38.256560+0000","last_undegraded":"2024-03-26T21:35:38.256560+0000","last_fullsized":"2024-03-26T21:35:38.256560+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:37.291480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:37.291480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:57.013417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":269,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259613+0000","last_change":"2024-03-26T21:34:23.051204+0000","last_active":"2024-03-26T21:35:38.259613+0000","last_peered":"2024-03-26T21:35:38.259613+0000","last_clean":"2024-03-26T21:35:38.259613+0000","last_became_active":"2024-03-26T21:32:55.469690+0000","last_became_peered":"2024-03-26T21:32:55.469690+0000","last_unstale":"2024-03-26T21:35:38.259613+0000","last_undegraded":"2024-03-26T21:35:38.259613+0000","last_fullsized":"2024-03-26T21:35:38.259613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:23.051104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:23.051104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.773972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":277,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638052+0000","last_change":"2024-03-26T21:35:33.934130+0000","last_active":"2024-03-26T21:35:41.638052+0000","last_peered":"2024-03-26T21:35:41.638052+0000","last_clean":"2024-03-26T21:35:41.638052+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:35:41.638052+0000","last_undegraded":"2024-03-26T21:35:41.638052+0000","last_fullsized":"2024-03-26T21:35:41.638052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:33.934014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.934014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:03.867388+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":272,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636015+0000","last_change":"2024-03-26T21:34:25.343996+0000","last_active":"2024-03-26T21:35:41.636015+0000","last_peered":"2024-03-26T21:35:41.636015+0000","last_clean":"2024-03-26T21:35:41.636015+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:35:41.636015+0000","last_undegraded":"2024-03-26T21:35:41.636015+0000","last_fullsized":"2024-03-26T21:35:41.636015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:34:25.343633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.343633+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":250,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636730+0000","last_change":"2024-03-26T21:32:29.393768+0000","last_active":"2024-03-26T21:35:41.636730+0000","last_peered":"2024-03-26T21:35:41.636730+0000","last_clean":"2024-03-26T21:35:41.636730+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:35:41.636730+0000","last_undegraded":"2024-03-26T21:35:41.636730+0000","last_fullsized":"2024-03-26T21:35:41.636730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:29.393601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:32:29.393601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":261,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257241+0000","last_change":"2024-03-26T21:35:14.435344+0000","last_active":"2024-03-26T21:35:38.257241+0000","last_peered":"2024-03-26T21:35:38.257241+0000","last_clean":"2024-03-26T21:35:38.257241+0000","last_became_active":"2024-03-26T21:32:55.474210+0000","last_became_peered":"2024-03-26T21:32:55.474210+0000","last_unstale":"2024-03-26T21:35:38.257241+0000","last_undegraded":"2024-03-26T21:35:38.257241+0000","last_fullsized":"2024-03-26T21:35:38.257241+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.435219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.435219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.974188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252955+0000","last_change":"2024-03-26T21:33:30.764236+0000","last_active":"2024-03-26T21:35:38.252955+0000","last_peered":"2024-03-26T21:35:38.252955+0000","last_clean":"2024-03-26T21:35:38.252955+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:35:38.252955+0000","last_undegraded":"2024-03-26T21:35:38.252955+0000","last_fullsized":"2024-03-26T21:35:38.252955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:30.763924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:33:30.763924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":285,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636320+0000","last_change":"2024-03-26T21:35:22.254470+0000","last_active":"2024-03-26T21:35:41.636320+0000","last_peered":"2024-03-26T21:35:41.636320+0000","last_clean":"2024-03-26T21:35:41.636320+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:35:41.636320+0000","last_undegraded":"2024-03-26T21:35:41.636320+0000","last_fullsized":"2024-03-26T21:35:41.636320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:35:22.254300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:22.254300+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:46.043692+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":250,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.634823+0000","last_change":"2024-03-26T21:32:26.906415+0000","last_active":"2024-03-26T21:35:41.634823+0000","last_peered":"2024-03-26T21:35:41.634823+0000","last_clean":"2024-03-26T21:35:41.634823+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:35:41.634823+0000","last_undegraded":"2024-03-26T21:35:41.634823+0000","last_fullsized":"2024-03-26T21:35:41.634823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:26.906307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:26.906307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":262,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636533+0000","last_change":"2024-03-26T21:35:19.231094+0000","last_active":"2024-03-26T21:35:41.636533+0000","last_peered":"2024-03-26T21:35:41.636533+0000","last_clean":"2024-03-26T21:35:41.636533+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:35:41.636533+0000","last_undegraded":"2024-03-26T21:35:41.636533+0000","last_fullsized":"2024-03-26T21:35:41.636533+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.230859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.230859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:42.696009+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256967+0000","last_change":"2024-03-26T21:35:26.578607+0000","last_active":"2024-03-26T21:35:38.256967+0000","last_peered":"2024-03-26T21:35:38.256967+0000","last_clean":"2024-03-26T21:35:38.256967+0000","last_became_active":"2024-03-26T21:32:55.470351+0000","last_became_peered":"2024-03-26T21:32:55.470351+0000","last_unstale":"2024-03-26T21:35:38.256967+0000","last_undegraded":"2024-03-26T21:35:38.256967+0000","last_fullsized":"2024-03-26T21:35:38.256967+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:26.578385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:26.578385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:49.039108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266910+0000","last_change":"2024-03-26T21:34:13.533465+0000","last_active":"2024-03-26T21:35:38.266910+0000","last_peered":"2024-03-26T21:35:38.266910+0000","last_clean":"2024-03-26T21:35:38.266910+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:35:38.266910+0000","last_undegraded":"2024-03-26T21:35:38.266910+0000","last_fullsized":"2024-03-26T21:35:38.266910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:34:13.533314+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:30:10.096107+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.533314+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:38.625607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636026+0000","last_change":"2024-03-26T21:35:13.297009+0000","last_active":"2024-03-26T21:35:41.636026+0000","last_peered":"2024-03-26T21:35:41.636026+0000","last_clean":"2024-03-26T21:35:41.636026+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:35:41.636026+0000","last_undegraded":"2024-03-26T21:35:41.636026+0000","last_fullsized":"2024-03-26T21:35:41.636026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.296881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:28.501742+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266695+0000","last_change":"2024-03-26T21:34:25.554360+0000","last_active":"2024-03-26T21:35:38.266695+0000","last_peered":"2024-03-26T21:35:38.266695+0000","last_clean":"2024-03-26T21:35:38.266695+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:35:38.266695+0000","last_undegraded":"2024-03-26T21:35:38.266695+0000","last_fullsized":"2024-03-26T21:35:38.266695+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.554230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.554230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":187,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637253+0000","last_change":"2024-03-26T21:35:28.149056+0000","last_active":"2024-03-26T21:35:41.637253+0000","last_peered":"2024-03-26T21:35:41.637253+0000","last_clean":"2024-03-26T21:35:41.637253+0000","last_became_active":"2024-03-26T21:32:55.467375+0000","last_became_peered":"2024-03-26T21:32:55.467375+0000","last_unstale":"2024-03-26T21:35:41.637253+0000","last_undegraded":"2024-03-26T21:35:41.637253+0000","last_fullsized":"2024-03-26T21:35:41.637253+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:28.148936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.148936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:55.503851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":160,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637125+0000","last_change":"2024-03-26T21:34:10.824961+0000","last_active":"2024-03-26T21:35:41.637125+0000","last_peered":"2024-03-26T21:35:41.637125+0000","last_clean":"2024-03-26T21:35:41.637125+0000","last_became_active":"2024-03-26T21:32:55.477977+0000","last_became_peered":"2024-03-26T21:32:55.477977+0000","last_unstale":"2024-03-26T21:35:41.637125+0000","last_undegraded":"2024-03-26T21:35:41.637125+0000","last_fullsized":"2024-03-26T21:35:41.637125+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:34:10.824829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.824829+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637502+0000","last_change":"2024-03-26T21:35:11.986871+0000","last_active":"2024-03-26T21:35:41.637502+0000","last_peered":"2024-03-26T21:35:41.637502+0000","last_clean":"2024-03-26T21:35:41.637502+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:35:41.637502+0000","last_undegraded":"2024-03-26T21:35:41.637502+0000","last_fullsized":"2024-03-26T21:35:41.637502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:35:11.986741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.986741+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:28.662140+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":354,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253118+0000","last_change":"2024-03-26T21:34:11.674348+0000","last_active":"2024-03-26T21:35:38.253118+0000","last_peered":"2024-03-26T21:35:38.253118+0000","last_clean":"2024-03-26T21:35:38.253118+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:35:38.253118+0000","last_undegraded":"2024-03-26T21:35:38.253118+0000","last_fullsized":"2024-03-26T21:35:38.253118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:34:11.674203+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:34:11.674203+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256547+0000","last_change":"2024-03-26T21:35:19.470998+0000","last_active":"2024-03-26T21:35:38.256547+0000","last_peered":"2024-03-26T21:35:38.256547+0000","last_clean":"2024-03-26T21:35:38.256547+0000","last_became_active":"2024-03-26T21:32:55.473442+0000","last_became_peered":"2024-03-26T21:32:55.473442+0000","last_unstale":"2024-03-26T21:35:38.256547+0000","last_undegraded":"2024-03-26T21:35:38.256547+0000","last_fullsized":"2024-03-26T21:35:38.256547+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.470873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.470873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.799000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"166'15","reported_seq":286,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638315+0000","last_change":"2024-03-26T21:34:27.404044+0000","last_active":"2024-03-26T21:35:41.638315+0000","last_peered":"2024-03-26T21:35:41.638315+0000","last_clean":"2024-03-26T21:35:41.638315+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:35:41.638315+0000","last_undegraded":"2024-03-26T21:35:41.638315+0000","last_fullsized":"2024-03-26T21:35:41.638315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:34:27.403785+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":175,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636919+0000","last_change":"2024-03-26T21:35:31.255202+0000","last_active":"2024-03-26T21:35:41.636919+0000","last_peered":"2024-03-26T21:35:41.636919+0000","last_clean":"2024-03-26T21:35:41.636919+0000","last_became_active":"2024-03-26T21:32:55.465075+0000","last_became_peered":"2024-03-26T21:32:55.465075+0000","last_unstale":"2024-03-26T21:35:41.636919+0000","last_undegraded":"2024-03-26T21:35:41.636919+0000","last_fullsized":"2024-03-26T21:35:41.636919+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:31.255065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:31.255065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:59.296575+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":359,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638502+0000","last_change":"2024-03-26T21:35:40.415843+0000","last_active":"2024-03-26T21:35:41.638502+0000","last_peered":"2024-03-26T21:35:41.638502+0000","last_clean":"2024-03-26T21:35:41.638502+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:35:41.638502+0000","last_undegraded":"2024-03-26T21:35:41.638502+0000","last_fullsized":"2024-03-26T21:35:41.638502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:35:40.415587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:40.415587+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:44.540436+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637095+0000","last_change":"2024-03-26T21:35:31.926557+0000","last_active":"2024-03-26T21:35:41.637095+0000","last_peered":"2024-03-26T21:35:41.637095+0000","last_clean":"2024-03-26T21:35:41.637095+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:35:41.637095+0000","last_undegraded":"2024-03-26T21:35:41.637095+0000","last_fullsized":"2024-03-26T21:35:41.637095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:31.926437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:31.926437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:41.602435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"146'13","reported_seq":291,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253324+0000","last_change":"2024-03-26T21:35:09.889342+0000","last_active":"2024-03-26T21:35:38.253324+0000","last_peered":"2024-03-26T21:35:38.253324+0000","last_clean":"2024-03-26T21:35:38.253324+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:35:38.253324+0000","last_undegraded":"2024-03-26T21:35:38.253324+0000","last_fullsized":"2024-03-26T21:35:38.253324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:35:09.889159+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.889159+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.850380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":266,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636472+0000","last_change":"2024-03-26T21:35:24.088893+0000","last_active":"2024-03-26T21:35:41.636472+0000","last_peered":"2024-03-26T21:35:41.636472+0000","last_clean":"2024-03-26T21:35:41.636472+0000","last_became_active":"2024-03-26T21:32:55.468099+0000","last_became_peered":"2024-03-26T21:32:55.468099+0000","last_unstale":"2024-03-26T21:35:41.636472+0000","last_undegraded":"2024-03-26T21:35:41.636472+0000","last_fullsized":"2024-03-26T21:35:41.636472+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:24.088758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:24.088758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:35.506081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":261,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.646654+0000","last_change":"2024-03-26T21:35:36.335634+0000","last_active":"2024-03-26T21:35:41.646654+0000","last_peered":"2024-03-26T21:35:41.646654+0000","last_clean":"2024-03-26T21:35:41.646654+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:35:41.646654+0000","last_undegraded":"2024-03-26T21:35:41.646654+0000","last_fullsized":"2024-03-26T21:35:41.646654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:36.335509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:36.335509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:03.885697+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257093+0000","last_change":"2024-03-26T21:35:21.539939+0000","last_active":"2024-03-26T21:35:38.257093+0000","last_peered":"2024-03-26T21:35:38.257093+0000","last_clean":"2024-03-26T21:35:38.257093+0000","last_became_active":"2024-03-26T21:32:55.470296+0000","last_became_peered":"2024-03-26T21:32:55.470296+0000","last_unstale":"2024-03-26T21:35:38.257093+0000","last_undegraded":"2024-03-26T21:35:38.257093+0000","last_fullsized":"2024-03-26T21:35:38.257093+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:21.539821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:21.539821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:37.230060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":285,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637910+0000","last_change":"2024-03-26T21:35:08.078326+0000","last_active":"2024-03-26T21:35:41.637910+0000","last_peered":"2024-03-26T21:35:41.637910+0000","last_clean":"2024-03-26T21:35:41.637910+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:35:41.637910+0000","last_undegraded":"2024-03-26T21:35:41.637910+0000","last_fullsized":"2024-03-26T21:35:41.637910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:35:08.078238+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.078238+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.504484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.290925+0000","last_change":"2024-03-26T21:35:38.290962+0000","last_active":"2024-03-26T21:35:38.290925+0000","last_peered":"2024-03-26T21:35:38.290925+0000","last_clean":"2024-03-26T21:35:38.290925+0000","last_became_active":"2024-03-26T21:32:55.470290+0000","last_became_peered":"2024-03-26T21:32:55.470290+0000","last_unstale":"2024-03-26T21:35:38.290925+0000","last_undegraded":"2024-03-26T21:35:38.290925+0000","last_fullsized":"2024-03-26T21:35:38.290925+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:38.290837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.290837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:58.043442+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":267,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253175+0000","last_change":"2024-03-26T21:35:30.820470+0000","last_active":"2024-03-26T21:35:38.253175+0000","last_peered":"2024-03-26T21:35:38.253175+0000","last_clean":"2024-03-26T21:35:38.253175+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:35:38.253175+0000","last_undegraded":"2024-03-26T21:35:38.253175+0000","last_fullsized":"2024-03-26T21:35:38.253175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:30.820356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:30.820356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:47.355485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.258741+0000","last_change":"2024-03-26T21:35:12.106142+0000","last_active":"2024-03-26T21:35:38.258741+0000","last_peered":"2024-03-26T21:35:38.258741+0000","last_clean":"2024-03-26T21:35:38.258741+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:35:38.258741+0000","last_undegraded":"2024-03-26T21:35:38.258741+0000","last_fullsized":"2024-03-26T21:35:38.258741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:12.106061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.106061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:38.680498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":288,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.646446+0000","last_change":"2024-03-26T21:35:38.322099+0000","last_active":"2024-03-26T21:35:41.646446+0000","last_peered":"2024-03-26T21:35:41.646446+0000","last_clean":"2024-03-26T21:35:41.646446+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:35:41.646446+0000","last_undegraded":"2024-03-26T21:35:41.646446+0000","last_fullsized":"2024-03-26T21:35:41.646446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:35:38.321844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.321844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:58.910057+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"182'3867","reported_seq":3895,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.362588+0000","last_change":"2024-03-26T21:34:58.693347+0000","last_active":"2024-03-26T21:35:41.362588+0000","last_peered":"2024-03-26T21:35:41.362588+0000","last_clean":"2024-03-26T21:35:41.362588+0000","last_became_active":"2024-03-26T21:34:58.692956+0000","last_became_peered":"2024-03-26T21:34:58.692956+0000","last_unstale":"2024-03-26T21:35:41.362588+0000","last_undegraded":"2024-03-26T21:35:41.362588+0000","last_fullsized":"2024-03-26T21:35:41.362588+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.842288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":11601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3867,"num_write_kb":247488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253756+0000","last_change":"2024-03-26T21:35:19.938802+0000","last_active":"2024-03-26T21:35:38.253756+0000","last_peered":"2024-03-26T21:35:38.253756+0000","last_clean":"2024-03-26T21:35:38.253756+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:35:38.253756+0000","last_undegraded":"2024-03-26T21:35:38.253756+0000","last_fullsized":"2024-03-26T21:35:38.253756+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.938637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.938637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.681951+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":287,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257209+0000","last_change":"2024-03-26T21:34:28.712452+0000","last_active":"2024-03-26T21:35:38.257209+0000","last_peered":"2024-03-26T21:35:38.257209+0000","last_clean":"2024-03-26T21:35:38.257209+0000","last_became_active":"2024-03-26T21:32:55.475860+0000","last_became_peered":"2024-03-26T21:32:55.475860+0000","last_unstale":"2024-03-26T21:35:38.257209+0000","last_undegraded":"2024-03-26T21:35:38.257209+0000","last_fullsized":"2024-03-26T21:35:38.257209+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:34:28.712368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.712368+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.654006+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":289,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252619+0000","last_change":"2024-03-26T21:34:32.761556+0000","last_active":"2024-03-26T21:35:38.252619+0000","last_peered":"2024-03-26T21:35:38.252619+0000","last_clean":"2024-03-26T21:35:38.252619+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:35:38.252619+0000","last_undegraded":"2024-03-26T21:35:38.252619+0000","last_fullsized":"2024-03-26T21:35:38.252619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:32.761425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.761425+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.953919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"134'10","reported_seq":357,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257290+0000","last_change":"2024-03-26T21:34:30.686544+0000","last_active":"2024-03-26T21:35:38.257290+0000","last_peered":"2024-03-26T21:35:38.257290+0000","last_clean":"2024-03-26T21:35:38.257290+0000","last_became_active":"2024-03-26T21:32:55.471127+0000","last_became_peered":"2024-03-26T21:32:55.471127+0000","last_unstale":"2024-03-26T21:35:38.257290+0000","last_undegraded":"2024-03-26T21:35:38.257290+0000","last_fullsized":"2024-03-26T21:35:38.257290+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:30.686411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:30.686411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.859040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"182'3852","reported_seq":3880,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.375892+0000","last_change":"2024-03-26T21:34:59.372364+0000","last_active":"2024-03-26T21:35:41.375892+0000","last_peered":"2024-03-26T21:35:41.375892+0000","last_clean":"2024-03-26T21:35:41.375892+0000","last_became_active":"2024-03-26T21:34:59.371915+0000","last_became_peered":"2024-03-26T21:34:59.371915+0000","last_unstale":"2024-03-26T21:35:41.375892+0000","last_undegraded":"2024-03-26T21:35:41.375892+0000","last_fullsized":"2024-03-26T21:35:41.375892+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:08.085476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":246528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":250,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636596+0000","last_change":"2024-03-26T21:32:42.225753+0000","last_active":"2024-03-26T21:35:41.636596+0000","last_peered":"2024-03-26T21:35:41.636596+0000","last_clean":"2024-03-26T21:35:41.636596+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:35:41.636596+0000","last_undegraded":"2024-03-26T21:35:41.636596+0000","last_fullsized":"2024-03-26T21:35:41.636596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:32:42.225645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:32:42.225645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257222+0000","last_change":"2024-03-26T21:35:15.464143+0000","last_active":"2024-03-26T21:35:38.257222+0000","last_peered":"2024-03-26T21:35:38.257222+0000","last_clean":"2024-03-26T21:35:38.257222+0000","last_became_active":"2024-03-26T21:32:55.471766+0000","last_became_peered":"2024-03-26T21:32:55.471766+0000","last_unstale":"2024-03-26T21:35:38.257222+0000","last_undegraded":"2024-03-26T21:35:38.257222+0000","last_fullsized":"2024-03-26T21:35:38.257222+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:35:15.464062+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.464062+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:29.495127+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":199,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636529+0000","last_change":"2024-03-26T21:34:18.762352+0000","last_active":"2024-03-26T21:35:41.636529+0000","last_peered":"2024-03-26T21:35:41.636529+0000","last_clean":"2024-03-26T21:35:41.636529+0000","last_became_active":"2024-03-26T21:32:55.467883+0000","last_became_peered":"2024-03-26T21:32:55.467883+0000","last_unstale":"2024-03-26T21:35:41.636529+0000","last_undegraded":"2024-03-26T21:35:41.636529+0000","last_fullsized":"2024-03-26T21:35:41.636529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:34:18.762259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.762259+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:44.929171+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.634831+0000","last_change":"2024-03-26T21:35:37.029521+0000","last_active":"2024-03-26T21:35:41.634831+0000","last_peered":"2024-03-26T21:35:41.634831+0000","last_clean":"2024-03-26T21:35:41.634831+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:35:41.634831+0000","last_undegraded":"2024-03-26T21:35:41.634831+0000","last_fullsized":"2024-03-26T21:35:41.634831+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:37.029395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:37.029395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:50.772745+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"183'3927","reported_seq":3957,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.927089+0000","last_change":"2024-03-26T21:34:59.371731+0000","last_active":"2024-03-26T21:35:41.927089+0000","last_peered":"2024-03-26T21:35:41.927089+0000","last_clean":"2024-03-26T21:35:41.927089+0000","last_became_active":"2024-03-26T21:34:59.371417+0000","last_became_peered":"2024-03-26T21:34:59.371417+0000","last_unstale":"2024-03-26T21:35:41.927089+0000","last_undegraded":"2024-03-26T21:35:41.927089+0000","last_fullsized":"2024-03-26T21:35:41.927089+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3927,"log_dups_size":0,"ondisk_log_size":3927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:12.074404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257359872,"num_objects":3927,"num_object_clones":0,"num_object_copies":11781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3927,"num_write_kb":251328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256519+0000","last_change":"2024-03-26T21:34:08.816123+0000","last_active":"2024-03-26T21:35:38.256519+0000","last_peered":"2024-03-26T21:35:38.256519+0000","last_clean":"2024-03-26T21:35:38.256519+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:35:38.256519+0000","last_undegraded":"2024-03-26T21:35:38.256519+0000","last_fullsized":"2024-03-26T21:35:38.256519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:08.816046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:08.816046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":278,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635279+0000","last_change":"2024-03-26T21:35:23.046553+0000","last_active":"2024-03-26T21:35:41.635279+0000","last_peered":"2024-03-26T21:35:41.635279+0000","last_clean":"2024-03-26T21:35:41.635279+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:35:41.635279+0000","last_undegraded":"2024-03-26T21:35:41.635279+0000","last_fullsized":"2024-03-26T21:35:41.635279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:23.046435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:23.046435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:35.412387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257090+0000","last_change":"2024-03-26T21:34:26.757750+0000","last_active":"2024-03-26T21:35:38.257090+0000","last_peered":"2024-03-26T21:35:38.257090+0000","last_clean":"2024-03-26T21:35:38.257090+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:35:38.257090+0000","last_undegraded":"2024-03-26T21:35:38.257090+0000","last_fullsized":"2024-03-26T21:35:38.257090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:26.757669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:26.757669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:56.675099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":312,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259078+0000","last_change":"2024-03-26T21:35:09.151841+0000","last_active":"2024-03-26T21:35:38.259078+0000","last_peered":"2024-03-26T21:35:38.259078+0000","last_clean":"2024-03-26T21:35:38.259078+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:35:38.259078+0000","last_undegraded":"2024-03-26T21:35:38.259078+0000","last_fullsized":"2024-03-26T21:35:38.259078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:35:09.151698+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.151698+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.490492+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"182'3964","reported_seq":3992,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.599824+0000","last_change":"2024-03-26T21:34:58.694137+0000","last_active":"2024-03-26T21:35:41.599824+0000","last_peered":"2024-03-26T21:35:41.599824+0000","last_clean":"2024-03-26T21:35:41.599824+0000","last_became_active":"2024-03-26T21:34:58.693834+0000","last_became_peered":"2024-03-26T21:34:58.693834+0000","last_unstale":"2024-03-26T21:35:41.599824+0000","last_undegraded":"2024-03-26T21:35:41.599824+0000","last_fullsized":"2024-03-26T21:35:41.599824+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3964,"log_dups_size":0,"ondisk_log_size":3964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.096424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259784704,"num_objects":3964,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3964,"num_write_kb":253696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636072+0000","last_change":"2024-03-26T21:35:13.040636+0000","last_active":"2024-03-26T21:35:41.636072+0000","last_peered":"2024-03-26T21:35:41.636072+0000","last_clean":"2024-03-26T21:35:41.636072+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:35:41.636072+0000","last_undegraded":"2024-03-26T21:35:41.636072+0000","last_fullsized":"2024-03-26T21:35:41.636072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.040364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.040364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.367879+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":271,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635805+0000","last_change":"2024-03-26T21:35:18.075830+0000","last_active":"2024-03-26T21:35:41.635805+0000","last_peered":"2024-03-26T21:35:41.635805+0000","last_clean":"2024-03-26T21:35:41.635805+0000","last_became_active":"2024-03-26T21:32:55.471707+0000","last_became_peered":"2024-03-26T21:32:55.471707+0000","last_unstale":"2024-03-26T21:35:41.635805+0000","last_undegraded":"2024-03-26T21:35:41.635805+0000","last_fullsized":"2024-03-26T21:35:41.635805+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:18.075661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:18.075661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:31.784375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253033+0000","last_change":"2024-03-26T21:34:14.695183+0000","last_active":"2024-03-26T21:35:38.253033+0000","last_peered":"2024-03-26T21:35:38.253033+0000","last_clean":"2024-03-26T21:35:38.253033+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:35:38.253033+0000","last_undegraded":"2024-03-26T21:35:38.253033+0000","last_fullsized":"2024-03-26T21:35:38.253033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.695081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.695081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":275,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638801+0000","last_change":"2024-03-26T21:34:15.948871+0000","last_active":"2024-03-26T21:35:41.638801+0000","last_peered":"2024-03-26T21:35:41.638801+0000","last_clean":"2024-03-26T21:35:41.638801+0000","last_became_active":"2024-03-26T21:32:55.478663+0000","last_became_peered":"2024-03-26T21:32:55.478663+0000","last_unstale":"2024-03-26T21:35:41.638801+0000","last_undegraded":"2024-03-26T21:35:41.638801+0000","last_fullsized":"2024-03-26T21:35:41.638801+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:34:15.948730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.948730+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"182'3892","reported_seq":3919,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.319158+0000","last_change":"2024-03-26T21:34:59.371815+0000","last_active":"2024-03-26T21:35:41.319158+0000","last_peered":"2024-03-26T21:35:41.319158+0000","last_clean":"2024-03-26T21:35:41.319158+0000","last_became_active":"2024-03-26T21:34:59.371434+0000","last_became_peered":"2024-03-26T21:34:59.371434+0000","last_unstale":"2024-03-26T21:35:41.319158+0000","last_undegraded":"2024-03-26T21:35:41.319158+0000","last_fullsized":"2024-03-26T21:35:41.319158+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3892,"log_dups_size":0,"ondisk_log_size":3892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.142848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255066112,"num_objects":3892,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":249088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":259,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637476+0000","last_change":"2024-03-26T21:34:10.292343+0000","last_active":"2024-03-26T21:35:41.637476+0000","last_peered":"2024-03-26T21:35:41.637476+0000","last_clean":"2024-03-26T21:35:41.637476+0000","last_became_active":"2024-03-26T21:32:55.468927+0000","last_became_peered":"2024-03-26T21:32:55.468927+0000","last_unstale":"2024-03-26T21:35:41.637476+0000","last_undegraded":"2024-03-26T21:35:41.637476+0000","last_fullsized":"2024-03-26T21:35:41.637476+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:10.292229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.292229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253620+0000","last_change":"2024-03-26T21:34:16.662556+0000","last_active":"2024-03-26T21:35:38.253620+0000","last_peered":"2024-03-26T21:35:38.253620+0000","last_clean":"2024-03-26T21:35:38.253620+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:35:38.253620+0000","last_undegraded":"2024-03-26T21:35:38.253620+0000","last_fullsized":"2024-03-26T21:35:38.253620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:34:16.662471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:16.662471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":186,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636685+0000","last_change":"2024-03-26T21:35:12.163203+0000","last_active":"2024-03-26T21:35:41.636685+0000","last_peered":"2024-03-26T21:35:41.636685+0000","last_clean":"2024-03-26T21:35:41.636685+0000","last_became_active":"2024-03-26T21:32:55.480439+0000","last_became_peered":"2024-03-26T21:32:55.480439+0000","last_unstale":"2024-03-26T21:35:41.636685+0000","last_undegraded":"2024-03-26T21:35:41.636685+0000","last_fullsized":"2024-03-26T21:35:41.636685+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:12.163075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.163075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.583097+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":352,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252846+0000","last_change":"2024-03-26T21:34:13.674583+0000","last_active":"2024-03-26T21:35:38.252846+0000","last_peered":"2024-03-26T21:35:38.252846+0000","last_clean":"2024-03-26T21:35:38.252846+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:35:38.252846+0000","last_undegraded":"2024-03-26T21:35:38.252846+0000","last_fullsized":"2024-03-26T21:35:38.252846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:34:13.674406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:13.674406+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":263,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637538+0000","last_change":"2024-03-26T21:34:18.955506+0000","last_active":"2024-03-26T21:35:41.637538+0000","last_peered":"2024-03-26T21:35:41.637538+0000","last_clean":"2024-03-26T21:35:41.637538+0000","last_became_active":"2024-03-26T21:32:55.478576+0000","last_became_peered":"2024-03-26T21:32:55.478576+0000","last_unstale":"2024-03-26T21:35:41.637538+0000","last_undegraded":"2024-03-26T21:35:41.637538+0000","last_fullsized":"2024-03-26T21:35:41.637538+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:18.955400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:18.955400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:45.675127+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"183'3946","reported_seq":3976,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:42.040868+0000","last_change":"2024-03-26T21:34:58.692851+0000","last_active":"2024-03-26T21:35:42.040868+0000","last_peered":"2024-03-26T21:35:42.040868+0000","last_clean":"2024-03-26T21:35:42.040868+0000","last_became_active":"2024-03-26T21:34:58.692166+0000","last_became_peered":"2024-03-26T21:34:58.692166+0000","last_unstale":"2024-03-26T21:35:42.040868+0000","last_undegraded":"2024-03-26T21:35:42.040868+0000","last_fullsized":"2024-03-26T21:35:42.040868+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:01.583848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258605056,"num_objects":3946,"num_object_clones":0,"num_object_copies":11838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":252544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257302+0000","last_change":"2024-03-26T21:35:27.571293+0000","last_active":"2024-03-26T21:35:38.257302+0000","last_peered":"2024-03-26T21:35:38.257302+0000","last_clean":"2024-03-26T21:35:38.257302+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:35:38.257302+0000","last_undegraded":"2024-03-26T21:35:38.257302+0000","last_fullsized":"2024-03-26T21:35:38.257302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.571209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.571209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:43.178878+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266841+0000","last_change":"2024-03-26T21:34:24.599907+0000","last_active":"2024-03-26T21:35:38.266841+0000","last_peered":"2024-03-26T21:35:38.266841+0000","last_clean":"2024-03-26T21:35:38.266841+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:35:38.266841+0000","last_undegraded":"2024-03-26T21:35:38.266841+0000","last_fullsized":"2024-03-26T21:35:38.266841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.599655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:51.967423+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":277,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635224+0000","last_change":"2024-03-26T21:34:29.929261+0000","last_active":"2024-03-26T21:35:41.635224+0000","last_peered":"2024-03-26T21:35:41.635224+0000","last_clean":"2024-03-26T21:35:41.635224+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:35:41.635224+0000","last_undegraded":"2024-03-26T21:35:41.635224+0000","last_fullsized":"2024-03-26T21:35:41.635224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:34:29.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":296,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267433+0000","last_change":"2024-03-26T21:34:10.523681+0000","last_active":"2024-03-26T21:35:38.267433+0000","last_peered":"2024-03-26T21:35:38.267433+0000","last_clean":"2024-03-26T21:35:38.267433+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:35:38.267433+0000","last_undegraded":"2024-03-26T21:35:38.267433+0000","last_fullsized":"2024-03-26T21:35:38.267433+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:34:10.523547+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.523547+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":262,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635348+0000","last_change":"2024-03-26T21:34:32.868281+0000","last_active":"2024-03-26T21:35:41.635348+0000","last_peered":"2024-03-26T21:35:41.635348+0000","last_clean":"2024-03-26T21:35:41.635348+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:35:41.635348+0000","last_undegraded":"2024-03-26T21:35:41.635348+0000","last_fullsized":"2024-03-26T21:35:41.635348+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.868200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:58.530052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"182'3962","reported_seq":3990,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.348369+0000","last_change":"2024-03-26T21:34:58.687897+0000","last_active":"2024-03-26T21:35:41.348369+0000","last_peered":"2024-03-26T21:35:41.348369+0000","last_clean":"2024-03-26T21:35:41.348369+0000","last_became_active":"2024-03-26T21:34:58.687512+0000","last_became_peered":"2024-03-26T21:34:58.687512+0000","last_unstale":"2024-03-26T21:35:41.348369+0000","last_undegraded":"2024-03-26T21:35:41.348369+0000","last_fullsized":"2024-03-26T21:35:41.348369+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:11.604741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259653632,"num_objects":3962,"num_object_clones":0,"num_object_copies":11886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3962,"num_write_kb":253568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":257,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253293+0000","last_change":"2024-03-26T21:33:26.736579+0000","last_active":"2024-03-26T21:35:38.253293+0000","last_peered":"2024-03-26T21:35:38.253293+0000","last_clean":"2024-03-26T21:35:38.253293+0000","last_became_active":"2024-03-26T21:32:55.474113+0000","last_became_peered":"2024-03-26T21:32:55.474113+0000","last_unstale":"2024-03-26T21:35:38.253293+0000","last_undegraded":"2024-03-26T21:35:38.253293+0000","last_fullsized":"2024-03-26T21:35:38.253293+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:33:26.736456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:33:26.736456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":520,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256732+0000","last_change":"2024-03-26T21:34:17.683647+0000","last_active":"2024-03-26T21:35:38.256732+0000","last_peered":"2024-03-26T21:35:38.256732+0000","last_clean":"2024-03-26T21:35:38.256732+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:35:38.256732+0000","last_undegraded":"2024-03-26T21:35:38.256732+0000","last_fullsized":"2024-03-26T21:35:38.256732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:34:17.683557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:34:17.683557+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636587+0000","last_change":"2024-03-26T21:34:28.378904+0000","last_active":"2024-03-26T21:35:41.636587+0000","last_peered":"2024-03-26T21:35:41.636587+0000","last_clean":"2024-03-26T21:35:41.636587+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:35:41.636587+0000","last_undegraded":"2024-03-26T21:35:41.636587+0000","last_fullsized":"2024-03-26T21:35:41.636587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:28.378787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:28.378787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"134'26","reported_seq":391,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259486+0000","last_change":"2024-03-26T21:34:31.976293+0000","last_active":"2024-03-26T21:35:38.259486+0000","last_peered":"2024-03-26T21:35:38.259486+0000","last_clean":"2024-03-26T21:35:38.259486+0000","last_became_active":"2024-03-26T21:32:55.477984+0000","last_became_peered":"2024-03-26T21:32:55.477984+0000","last_unstale":"2024-03-26T21:35:38.259486+0000","last_undegraded":"2024-03-26T21:35:38.259486+0000","last_fullsized":"2024-03-26T21:35:38.259486+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'26","last_scrub_stamp":"2024-03-26T21:34:31.976190+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:34:31.976190+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:00.446068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.257729+0000","last_change":"2024-03-26T21:34:20.660945+0000","last_active":"2024-03-26T21:35:38.257729+0000","last_peered":"2024-03-26T21:35:38.257729+0000","last_clean":"2024-03-26T21:35:38.257729+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:35:38.257729+0000","last_undegraded":"2024-03-26T21:35:38.257729+0000","last_fullsized":"2024-03-26T21:35:38.257729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:20.660825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:20.660825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.066534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"182'3873","reported_seq":3901,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.611618+0000","last_change":"2024-03-26T21:34:58.693396+0000","last_active":"2024-03-26T21:35:41.611618+0000","last_peered":"2024-03-26T21:35:41.611618+0000","last_clean":"2024-03-26T21:35:41.611618+0000","last_became_active":"2024-03-26T21:34:58.693101+0000","last_became_peered":"2024-03-26T21:34:58.693101+0000","last_unstale":"2024-03-26T21:35:41.611618+0000","last_undegraded":"2024-03-26T21:35:41.611618+0000","last_fullsized":"2024-03-26T21:35:41.611618+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:17.317391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253820928,"num_objects":3873,"num_object_clones":0,"num_object_copies":11619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3873,"num_write_kb":247872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267220+0000","last_change":"2024-03-26T21:34:14.558500+0000","last_active":"2024-03-26T21:35:38.267220+0000","last_peered":"2024-03-26T21:35:38.267220+0000","last_clean":"2024-03-26T21:35:38.267220+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:35:38.267220+0000","last_undegraded":"2024-03-26T21:35:38.267220+0000","last_fullsized":"2024-03-26T21:35:38.267220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:14.557934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:14.557934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":271,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635866+0000","last_change":"2024-03-26T21:34:25.034122+0000","last_active":"2024-03-26T21:35:41.635866+0000","last_peered":"2024-03-26T21:35:41.635866+0000","last_clean":"2024-03-26T21:35:41.635866+0000","last_became_active":"2024-03-26T21:32:55.475773+0000","last_became_peered":"2024-03-26T21:32:55.475773+0000","last_unstale":"2024-03-26T21:35:41.635866+0000","last_undegraded":"2024-03-26T21:35:41.635866+0000","last_fullsized":"2024-03-26T21:35:41.635866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:25.033987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:25.033987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":706,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:39.901308+0000","last_change":"2024-03-26T21:35:11.460682+0000","last_active":"2024-03-26T21:35:39.901308+0000","last_peered":"2024-03-26T21:35:39.901308+0000","last_clean":"2024-03-26T21:35:39.901308+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:35:39.901308+0000","last_undegraded":"2024-03-26T21:35:39.901308+0000","last_fullsized":"2024-03-26T21:35:39.901308+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:35:11.460547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.460547+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.085903+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":420,"num_read_kb":415,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":304,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637612+0000","last_change":"2024-03-26T21:32:39.859396+0000","last_active":"2024-03-26T21:35:41.637612+0000","last_peered":"2024-03-26T21:35:41.637612+0000","last_clean":"2024-03-26T21:35:41.637612+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:35:41.637612+0000","last_undegraded":"2024-03-26T21:35:41.637612+0000","last_fullsized":"2024-03-26T21:35:41.637612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:32:39.858918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:32:39.858918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253452+0000","last_change":"2024-03-26T21:34:09.662284+0000","last_active":"2024-03-26T21:35:38.253452+0000","last_peered":"2024-03-26T21:35:38.253452+0000","last_clean":"2024-03-26T21:35:38.253452+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:35:38.253452+0000","last_undegraded":"2024-03-26T21:35:38.253452+0000","last_fullsized":"2024-03-26T21:35:38.253452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:09.662190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:09.662190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"183'3980","reported_seq":4010,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:42.048739+0000","last_change":"2024-03-26T21:34:58.693123+0000","last_active":"2024-03-26T21:35:42.048739+0000","last_peered":"2024-03-26T21:35:42.048739+0000","last_clean":"2024-03-26T21:35:42.048739+0000","last_became_active":"2024-03-26T21:34:58.692159+0000","last_became_peered":"2024-03-26T21:34:58.692159+0000","last_unstale":"2024-03-26T21:35:42.048739+0000","last_undegraded":"2024-03-26T21:35:42.048739+0000","last_fullsized":"2024-03-26T21:35:42.048739+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:08.426555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":254720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266569+0000","last_change":"2024-03-26T21:35:33.446019+0000","last_active":"2024-03-26T21:35:38.266569+0000","last_peered":"2024-03-26T21:35:38.266569+0000","last_clean":"2024-03-26T21:35:38.266569+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:35:38.266569+0000","last_undegraded":"2024-03-26T21:35:38.266569+0000","last_fullsized":"2024-03-26T21:35:38.266569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:33.445910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.445910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:38.212824+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267436+0000","last_change":"2024-03-26T21:35:17.424167+0000","last_active":"2024-03-26T21:35:38.267436+0000","last_peered":"2024-03-26T21:35:38.267436+0000","last_clean":"2024-03-26T21:35:38.267436+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:35:38.267436+0000","last_undegraded":"2024-03-26T21:35:38.267436+0000","last_fullsized":"2024-03-26T21:35:38.267436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:17.424050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:35:17.424050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:39.574813+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":302,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638656+0000","last_change":"2024-03-26T21:35:11.003728+0000","last_active":"2024-03-26T21:35:41.638656+0000","last_peered":"2024-03-26T21:35:41.638656+0000","last_clean":"2024-03-26T21:35:41.638656+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:35:41.638656+0000","last_undegraded":"2024-03-26T21:35:41.638656+0000","last_fullsized":"2024-03-26T21:35:41.638656+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:35:11.003580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.003580+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.140682+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.259116+0000","last_change":"2024-03-26T21:35:15.119981+0000","last_active":"2024-03-26T21:35:38.259116+0000","last_peered":"2024-03-26T21:35:38.259116+0000","last_clean":"2024-03-26T21:35:38.259116+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:35:38.259116+0000","last_undegraded":"2024-03-26T21:35:38.259116+0000","last_fullsized":"2024-03-26T21:35:38.259116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.119862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.119862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.753214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"183'3936","reported_seq":3966,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.941919+0000","last_change":"2024-03-26T21:34:58.689609+0000","last_active":"2024-03-26T21:35:41.941919+0000","last_peered":"2024-03-26T21:35:41.941919+0000","last_clean":"2024-03-26T21:35:41.941919+0000","last_became_active":"2024-03-26T21:34:58.689106+0000","last_became_peered":"2024-03-26T21:34:58.689106+0000","last_unstale":"2024-03-26T21:35:41.941919+0000","last_undegraded":"2024-03-26T21:35:41.941919+0000","last_fullsized":"2024-03-26T21:35:41.941919+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:26.882021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257949696,"num_objects":3936,"num_object_clones":0,"num_object_copies":11808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3936,"num_write_kb":251904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.634879+0000","last_change":"2024-03-26T21:35:30.917840+0000","last_active":"2024-03-26T21:35:41.634879+0000","last_peered":"2024-03-26T21:35:41.634879+0000","last_clean":"2024-03-26T21:35:41.634879+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:35:41.634879+0000","last_undegraded":"2024-03-26T21:35:41.634879+0000","last_fullsized":"2024-03-26T21:35:41.634879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:30.917720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:30.917720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:39.169718+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253178+0000","last_change":"2024-03-26T21:35:25.841787+0000","last_active":"2024-03-26T21:35:38.253178+0000","last_peered":"2024-03-26T21:35:38.253178+0000","last_clean":"2024-03-26T21:35:38.253178+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:35:38.253178+0000","last_undegraded":"2024-03-26T21:35:38.253178+0000","last_fullsized":"2024-03-26T21:35:38.253178+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.841708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.841708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:45.777516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":288,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.266806+0000","last_change":"2024-03-26T21:35:15.509109+0000","last_active":"2024-03-26T21:35:38.266806+0000","last_peered":"2024-03-26T21:35:38.266806+0000","last_clean":"2024-03-26T21:35:38.266806+0000","last_became_active":"2024-03-26T21:32:55.465412+0000","last_became_peered":"2024-03-26T21:32:55.465412+0000","last_unstale":"2024-03-26T21:35:38.266806+0000","last_undegraded":"2024-03-26T21:35:38.266806+0000","last_fullsized":"2024-03-26T21:35:38.266806+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:35:15.508751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.508751+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.773583+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":356,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638421+0000","last_change":"2024-03-26T21:35:28.921519+0000","last_active":"2024-03-26T21:35:41.638421+0000","last_peered":"2024-03-26T21:35:41.638421+0000","last_clean":"2024-03-26T21:35:41.638421+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:35:41.638421+0000","last_undegraded":"2024-03-26T21:35:41.638421+0000","last_fullsized":"2024-03-26T21:35:41.638421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:35:28.921392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.921392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:54.147492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"182'3822","reported_seq":3850,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.362496+0000","last_change":"2024-03-26T21:34:59.372041+0000","last_active":"2024-03-26T21:35:41.362496+0000","last_peered":"2024-03-26T21:35:41.362496+0000","last_clean":"2024-03-26T21:35:41.362496+0000","last_became_active":"2024-03-26T21:34:59.371634+0000","last_became_peered":"2024-03-26T21:34:59.371634+0000","last_unstale":"2024-03-26T21:35:41.362496+0000","last_undegraded":"2024-03-26T21:35:41.362496+0000","last_fullsized":"2024-03-26T21:35:41.362496+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:21.299560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250478592,"num_objects":3822,"num_object_clones":0,"num_object_copies":11466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3822,"num_write_kb":244608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.256723+0000","last_change":"2024-03-26T21:35:10.438550+0000","last_active":"2024-03-26T21:35:38.256723+0000","last_peered":"2024-03-26T21:35:38.256723+0000","last_clean":"2024-03-26T21:35:38.256723+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:35:38.256723+0000","last_undegraded":"2024-03-26T21:35:38.256723+0000","last_fullsized":"2024-03-26T21:35:38.256723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:10.438407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:10.438407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:30.156978+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":264,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637709+0000","last_change":"2024-03-26T21:34:10.057922+0000","last_active":"2024-03-26T21:35:41.637709+0000","last_peered":"2024-03-26T21:35:41.637709+0000","last_clean":"2024-03-26T21:35:41.637709+0000","last_became_active":"2024-03-26T21:32:55.479274+0000","last_became_peered":"2024-03-26T21:32:55.479274+0000","last_unstale":"2024-03-26T21:35:41.637709+0000","last_undegraded":"2024-03-26T21:35:41.637709+0000","last_fullsized":"2024-03-26T21:35:41.637709+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:34:10.057797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:10.057797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":203,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636451+0000","last_change":"2024-03-26T21:35:26.141597+0000","last_active":"2024-03-26T21:35:41.636451+0000","last_peered":"2024-03-26T21:35:41.636451+0000","last_clean":"2024-03-26T21:35:41.636451+0000","last_became_active":"2024-03-26T21:32:55.469472+0000","last_became_peered":"2024-03-26T21:32:55.469472+0000","last_unstale":"2024-03-26T21:35:41.636451+0000","last_undegraded":"2024-03-26T21:35:41.636451+0000","last_fullsized":"2024-03-26T21:35:41.636451+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:35:26.141418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:26.141418+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:49.710634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"134'10","reported_seq":359,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636224+0000","last_change":"2024-03-26T21:34:12.341709+0000","last_active":"2024-03-26T21:35:41.636224+0000","last_peered":"2024-03-26T21:35:41.636224+0000","last_clean":"2024-03-26T21:35:41.636224+0000","last_became_active":"2024-03-26T21:32:55.476581+0000","last_became_peered":"2024-03-26T21:32:55.476581+0000","last_unstale":"2024-03-26T21:35:41.636224+0000","last_undegraded":"2024-03-26T21:35:41.636224+0000","last_fullsized":"2024-03-26T21:35:41.636224+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:34:12.341310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:12.341310+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"183'3927","reported_seq":3957,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.930500+0000","last_change":"2024-03-26T21:34:59.370935+0000","last_active":"2024-03-26T21:35:41.930500+0000","last_peered":"2024-03-26T21:35:41.930500+0000","last_clean":"2024-03-26T21:35:41.930500+0000","last_became_active":"2024-03-26T21:34:59.370503+0000","last_became_peered":"2024-03-26T21:34:59.370503+0000","last_unstale":"2024-03-26T21:35:41.930500+0000","last_undegraded":"2024-03-26T21:35:41.930500+0000","last_fullsized":"2024-03-26T21:35:41.930500+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3927,"log_dups_size":0,"ondisk_log_size":3927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:21.453666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257359872,"num_objects":3927,"num_object_clones":0,"num_object_copies":11781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3927,"num_write_kb":251328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":168,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636682+0000","last_change":"2024-03-26T21:34:33.905393+0000","last_active":"2024-03-26T21:35:41.636682+0000","last_peered":"2024-03-26T21:35:41.636682+0000","last_clean":"2024-03-26T21:35:41.636682+0000","last_became_active":"2024-03-26T21:32:55.476227+0000","last_became_peered":"2024-03-26T21:32:55.476227+0000","last_unstale":"2024-03-26T21:35:41.636682+0000","last_undegraded":"2024-03-26T21:35:41.636682+0000","last_fullsized":"2024-03-26T21:35:41.636682+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.905276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.905276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:41.967842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.646195+0000","last_change":"2024-03-26T21:35:28.245063+0000","last_active":"2024-03-26T21:35:41.646195+0000","last_peered":"2024-03-26T21:35:41.646195+0000","last_clean":"2024-03-26T21:35:41.646195+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:35:41.646195+0000","last_undegraded":"2024-03-26T21:35:41.646195+0000","last_fullsized":"2024-03-26T21:35:41.646195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:28.244684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.244684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:41.966801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":298,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636312+0000","last_change":"2024-03-26T21:35:33.264772+0000","last_active":"2024-03-26T21:35:41.636312+0000","last_peered":"2024-03-26T21:35:41.636312+0000","last_clean":"2024-03-26T21:35:41.636312+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:35:41.636312+0000","last_undegraded":"2024-03-26T21:35:41.636312+0000","last_fullsized":"2024-03-26T21:35:41.636312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:35:33.264627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.264627+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:56.288197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":129,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636190+0000","last_change":"2024-03-26T21:34:07.791992+0000","last_active":"2024-03-26T21:35:41.636190+0000","last_peered":"2024-03-26T21:35:41.636190+0000","last_clean":"2024-03-26T21:35:41.636190+0000","last_became_active":"2024-03-26T21:32:55.468331+0000","last_became_peered":"2024-03-26T21:32:55.468331+0000","last_unstale":"2024-03-26T21:35:41.636190+0000","last_undegraded":"2024-03-26T21:35:41.636190+0000","last_fullsized":"2024-03-26T21:35:41.636190+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.791763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.791763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"182'3887","reported_seq":3915,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.613960+0000","last_change":"2024-03-26T21:34:58.691128+0000","last_active":"2024-03-26T21:35:41.613960+0000","last_peered":"2024-03-26T21:35:41.613960+0000","last_clean":"2024-03-26T21:35:41.613960+0000","last_became_active":"2024-03-26T21:34:58.690860+0000","last_became_peered":"2024-03-26T21:34:58.690860+0000","last_unstale":"2024-03-26T21:35:41.613960+0000","last_undegraded":"2024-03-26T21:35:41.613960+0000","last_fullsized":"2024-03-26T21:35:41.613960+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.930774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254738432,"num_objects":3887,"num_object_clones":0,"num_object_copies":11661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3887,"num_write_kb":248768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.252926+0000","last_change":"2024-03-26T21:34:15.695258+0000","last_active":"2024-03-26T21:35:38.252926+0000","last_peered":"2024-03-26T21:35:38.252926+0000","last_clean":"2024-03-26T21:35:38.252926+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:35:38.252926+0000","last_undegraded":"2024-03-26T21:35:38.252926+0000","last_fullsized":"2024-03-26T21:35:38.252926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:15.695126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:34:15.695126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"134'10","reported_seq":240,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637707+0000","last_change":"2024-03-26T21:35:09.138220+0000","last_active":"2024-03-26T21:35:41.637707+0000","last_peered":"2024-03-26T21:35:41.637707+0000","last_clean":"2024-03-26T21:35:41.637707+0000","last_became_active":"2024-03-26T21:32:55.478624+0000","last_became_peered":"2024-03-26T21:32:55.478624+0000","last_unstale":"2024-03-26T21:35:41.637707+0000","last_undegraded":"2024-03-26T21:35:41.637707+0000","last_fullsized":"2024-03-26T21:35:41.637707+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:35:09.138087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.138087+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:20.372198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":270,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636068+0000","last_change":"2024-03-26T21:35:14.322330+0000","last_active":"2024-03-26T21:35:41.636068+0000","last_peered":"2024-03-26T21:35:41.636068+0000","last_clean":"2024-03-26T21:35:41.636068+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:35:41.636068+0000","last_undegraded":"2024-03-26T21:35:41.636068+0000","last_fullsized":"2024-03-26T21:35:41.636068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.322126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.322126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:15.039978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":298,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.267283+0000","last_change":"2024-03-26T21:34:33.587083+0000","last_active":"2024-03-26T21:35:38.267283+0000","last_peered":"2024-03-26T21:35:38.267283+0000","last_clean":"2024-03-26T21:35:38.267283+0000","last_became_active":"2024-03-26T21:32:55.471751+0000","last_became_peered":"2024-03-26T21:32:55.471751+0000","last_unstale":"2024-03-26T21:35:38.267283+0000","last_undegraded":"2024-03-26T21:35:38.267283+0000","last_fullsized":"2024-03-26T21:35:38.267283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:34:33.586983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.586983+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"183'4111","reported_seq":4141,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:43.459460+0000","last_change":"2024-03-26T21:34:58.687626+0000","last_active":"2024-03-26T21:35:43.459460+0000","last_peered":"2024-03-26T21:35:43.459460+0000","last_clean":"2024-03-26T21:35:43.459460+0000","last_became_active":"2024-03-26T21:34:58.687260+0000","last_became_peered":"2024-03-26T21:34:58.687260+0000","last_unstale":"2024-03-26T21:35:43.459460+0000","last_undegraded":"2024-03-26T21:35:43.459460+0000","last_fullsized":"2024-03-26T21:35:43.459460+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.659671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269418496,"num_objects":4111,"num_object_clones":0,"num_object_copies":12333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4111,"num_write_kb":263104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":258,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.638044+0000","last_change":"2024-03-26T21:34:32.360553+0000","last_active":"2024-03-26T21:35:41.638044+0000","last_peered":"2024-03-26T21:35:41.638044+0000","last_clean":"2024-03-26T21:35:41.638044+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:35:41.638044+0000","last_undegraded":"2024-03-26T21:35:41.638044+0000","last_fullsized":"2024-03-26T21:35:41.638044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:32.360471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:34:32.360471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.149069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":261,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.253421+0000","last_change":"2024-03-26T21:35:08.865267+0000","last_active":"2024-03-26T21:35:38.253421+0000","last_peered":"2024-03-26T21:35:38.253421+0000","last_clean":"2024-03-26T21:35:38.253421+0000","last_became_active":"2024-03-26T21:32:55.465348+0000","last_became_peered":"2024-03-26T21:32:55.465348+0000","last_unstale":"2024-03-26T21:35:38.253421+0000","last_undegraded":"2024-03-26T21:35:38.253421+0000","last_fullsized":"2024-03-26T21:35:38.253421+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:08.865126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:08.865126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:24.367221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":180,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636333+0000","last_change":"2024-03-26T21:34:24.836203+0000","last_active":"2024-03-26T21:35:41.636333+0000","last_peered":"2024-03-26T21:35:41.636333+0000","last_clean":"2024-03-26T21:35:41.636333+0000","last_became_active":"2024-03-26T21:32:55.463332+0000","last_became_peered":"2024-03-26T21:32:55.463332+0000","last_unstale":"2024-03-26T21:35:41.636333+0000","last_undegraded":"2024-03-26T21:35:41.636333+0000","last_fullsized":"2024-03-26T21:35:41.636333+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:24.835878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:24.835878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":199,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637314+0000","last_change":"2024-03-26T21:35:20.104232+0000","last_active":"2024-03-26T21:35:41.637314+0000","last_peered":"2024-03-26T21:35:41.637314+0000","last_clean":"2024-03-26T21:35:41.637314+0000","last_became_active":"2024-03-26T21:32:55.471887+0000","last_became_peered":"2024-03-26T21:32:55.471887+0000","last_unstale":"2024-03-26T21:35:41.637314+0000","last_undegraded":"2024-03-26T21:35:41.637314+0000","last_fullsized":"2024-03-26T21:35:41.637314+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:35:20.104039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:20.104039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:42.567579+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"182'3925","reported_seq":3952,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.609983+0000","last_change":"2024-03-26T21:34:58.693459+0000","last_active":"2024-03-26T21:35:41.609983+0000","last_peered":"2024-03-26T21:35:41.609983+0000","last_clean":"2024-03-26T21:35:41.609983+0000","last_became_active":"2024-03-26T21:34:58.693174+0000","last_became_peered":"2024-03-26T21:34:58.693174+0000","last_unstale":"2024-03-26T21:35:41.609983+0000","last_undegraded":"2024-03-26T21:35:41.609983+0000","last_fullsized":"2024-03-26T21:35:41.609983+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:03.413097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257228800,"num_objects":3925,"num_object_clones":0,"num_object_copies":11775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":251200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":267,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635512+0000","last_change":"2024-03-26T21:35:32.053084+0000","last_active":"2024-03-26T21:35:41.635512+0000","last_peered":"2024-03-26T21:35:41.635512+0000","last_clean":"2024-03-26T21:35:41.635512+0000","last_became_active":"2024-03-26T21:32:55.470413+0000","last_became_peered":"2024-03-26T21:32:55.470413+0000","last_unstale":"2024-03-26T21:35:41.635512+0000","last_undegraded":"2024-03-26T21:35:41.635512+0000","last_fullsized":"2024-03-26T21:35:41.635512+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:32.052678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:35:32.052678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:41.197036+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:39.437728+0000","last_change":"2024-03-26T21:35:39.438265+0000","last_active":"2024-03-26T21:35:39.437728+0000","last_peered":"2024-03-26T21:35:39.437728+0000","last_clean":"2024-03-26T21:35:39.437728+0000","last_became_active":"2024-03-26T21:32:55.472013+0000","last_became_peered":"2024-03-26T21:32:55.472013+0000","last_unstale":"2024-03-26T21:35:39.437728+0000","last_undegraded":"2024-03-26T21:35:39.437728+0000","last_fullsized":"2024-03-26T21:35:39.437728+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:39.437635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:39.437635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:07.806010+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-26T21:35:38.258901+0000","last_change":"2024-03-26T21:34:33.006206+0000","last_active":"2024-03-26T21:35:38.258901+0000","last_peered":"2024-03-26T21:35:38.258901+0000","last_clean":"2024-03-26T21:35:38.258901+0000","last_became_active":"2024-03-26T21:32:55.469205+0000","last_became_peered":"2024-03-26T21:32:55.469205+0000","last_unstale":"2024-03-26T21:35:38.258901+0000","last_undegraded":"2024-03-26T21:35:38.258901+0000","last_fullsized":"2024-03-26T21:35:38.258901+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:34:33.006101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:35:50.004115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":211,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637264+0000","last_change":"2024-03-26T21:35:13.172081+0000","last_active":"2024-03-26T21:35:41.637264+0000","last_peered":"2024-03-26T21:35:41.637264+0000","last_clean":"2024-03-26T21:35:41.637264+0000","last_became_active":"2024-03-26T21:32:55.464394+0000","last_became_peered":"2024-03-26T21:32:55.464394+0000","last_unstale":"2024-03-26T21:35:41.637264+0000","last_undegraded":"2024-03-26T21:35:41.637264+0000","last_fullsized":"2024-03-26T21:35:41.637264+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:35:13.171864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.171864+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:36.108545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"183'3865","reported_seq":3895,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:42.009852+0000","last_change":"2024-03-26T21:34:58.692961+0000","last_active":"2024-03-26T21:35:42.009852+0000","last_peered":"2024-03-26T21:35:42.009852+0000","last_clean":"2024-03-26T21:35:42.009852+0000","last_became_active":"2024-03-26T21:34:58.692182+0000","last_became_peered":"2024-03-26T21:34:58.692182+0000","last_unstale":"2024-03-26T21:35:42.009852+0000","last_undegraded":"2024-03-26T21:35:42.009852+0000","last_fullsized":"2024-03-26T21:35:42.009852+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:14.418350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253296640,"num_objects":3865,"num_object_clones":0,"num_object_copies":11595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":247360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":172,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.637544+0000","last_change":"2024-03-26T21:35:15.211309+0000","last_active":"2024-03-26T21:35:41.637544+0000","last_peered":"2024-03-26T21:35:41.637544+0000","last_clean":"2024-03-26T21:35:41.637544+0000","last_became_active":"2024-03-26T21:32:55.471347+0000","last_became_peered":"2024-03-26T21:32:55.471347+0000","last_unstale":"2024-03-26T21:35:41.637544+0000","last_undegraded":"2024-03-26T21:35:41.637544+0000","last_fullsized":"2024-03-26T21:35:41.637544+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.211195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.211195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:36:18.809000+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":364,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.635852+0000","last_change":"2024-03-26T21:35:35.985426+0000","last_active":"2024-03-26T21:35:41.635852+0000","last_peered":"2024-03-26T21:35:41.635852+0000","last_clean":"2024-03-26T21:35:41.635852+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:35:41.635852+0000","last_undegraded":"2024-03-26T21:35:41.635852+0000","last_fullsized":"2024-03-26T21:35:41.635852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:35:35.985297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:35.985297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:00.039444+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":263,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:41.636399+0000","last_change":"2024-03-26T21:34:07.273064+0000","last_active":"2024-03-26T21:35:41.636399+0000","last_peered":"2024-03-26T21:35:41.636399+0000","last_clean":"2024-03-26T21:35:41.636399+0000","last_became_active":"2024-03-26T21:32:55.473354+0000","last_became_peered":"2024-03-26T21:32:55.473354+0000","last_unstale":"2024-03-26T21:35:41.636399+0000","last_undegraded":"2024-03-26T21:35:41.636399+0000","last_fullsized":"2024-03-26T21:35:41.636399+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:07.272948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:34:07.272948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":202,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-03-26T21:35:43.214888+0000","last_change":"2024-03-26T21:35:43.214936+0000","last_active":"2024-03-26T21:35:43.214888+0000","last_peered":"2024-03-26T21:35:43.214888+0000","last_clean":"2024-03-26T21:35:43.214888+0000","last_became_active":"2024-03-26T21:32:55.476367+0000","last_became_peered":"2024-03-26T21:32:55.476367+0000","last_unstale":"2024-03-26T21:35:43.214888+0000","last_undegraded":"2024-03-26T21:35:43.214888+0000","last_fullsized":"2024-03-26T21:35:43.214888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:35:43.214737+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:35:43.214737+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:37:10.046164+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4111466496,"num_objects":62736,"num_object_clones":0,"num_object_copies":188208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62736,"num_write_kb":4015104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12338528256,"data_stored":12338528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62736,"ondisk_log_size":62736,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":420,"num_read_kb":415,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070554,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383568,"kb_used_data":1235784,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92361584,"statfs":{"total":95995035648,"available":94578262016,"internally_reserved":0,"allocated":1265442816,"data_stored":1264995096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364896,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1392244,"kb_used_data":1244456,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92352908,"statfs":{"total":95995035648,"available":94569377792,"internally_reserved":0,"allocated":1274322944,"data_stored":1273905580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914144,"kb_used_data":1742744,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91831008,"statfs":{"total":95995035648,"available":94034952192,"internally_reserved":0,"allocated":1784569856,"data_stored":1784104659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738495,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1796960,"kb_used_data":1518676,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91948192,"statfs":{"total":95995035648,"available":94154948608,"internally_reserved":0,"allocated":1555124224,"data_stored":1554670379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1642216,"kb_used_data":1480544,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92102936,"statfs":{"total":95995035648,"available":94313406464,"internally_reserved":0,"allocated":1516077056,"data_stored":1515667455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1763412,"kb_used_data":1493452,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91981740,"statfs":{"total":95995035648,"available":94189301760,"internally_reserved":0,"allocated":1529294848,"data_stored":1528840980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650403,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483872,"kb_used_data":1293460,"kb_used_omap":4,"kb_used_meta":190395,"kb_avail":92261280,"statfs":{"total":95995035648,"available":94475550720,"internally_reserved":0,"allocated":1324503040,"data_stored":1324048885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":194965155},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659235,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2090368,"kb_used_data":1964792,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91654784,"statfs":{"total":95995035648,"available":93854498816,"internally_reserved":0,"allocated":2011947008,"data_stored":2011513400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1554251776,"data_stored":1554251776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1790640128,"data_stored":1790640128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1532428288,"data_stored":1532428288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1551695872,"data_stored":1551695872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1333329920,"data_stored":1333329920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2031353856,"data_stored":2031353856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1278803968,"data_stored":1278803968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1266024448,"data_stored":1266024448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:35:50.260 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T21:35:50.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:35:50.260 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T21:35:50.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd out 4 2024-03-26T21:35:50.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:50.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:50.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:50.436 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:50.538 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.591 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:50.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:50.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:50.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:50.733 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:35:50.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:50.879 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:50.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: pgmap v302: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2799044400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:50.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:50.972 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:50 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:35:50.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:50.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 42 16 74514 74498 110.836 80.8125 0.00830982 0.00901743 2024-03-26T21:35:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:51.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:51.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: pgmap v302: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:51.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2799044400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: pgmap v302: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2799044400' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:35:51.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:50 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:35:51.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:51.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:51.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:51.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:51.452 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:35:51.452 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:35:51.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.563 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:51.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:51.691 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:51.704 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:51.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:51.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[40587]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:35:51.897 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:51 smithi055 ceph-mon[29406]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:35:51.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:51.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 43 16 75833 75817 110.175 82.4375 0.00711477 0.00907183 2024-03-26T21:35:51.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:51 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:35:51.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:35:51.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:51 smithi067 ceph-mon[33346]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T21:35:51.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:51.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:52.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:52.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:52.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:35:52.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:52.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.447 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:52.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:35:52.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:52.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:52.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:52.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:35:52.880 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:52.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:52.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:52 smithi067 ceph-mon[33346]: pgmap v305: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:52.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:52.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 44 16 77171 77155 109.572 83.625 0.0211822 0.00912077 2024-03-26T21:35:53.005 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:52 smithi055 ceph-mon[29406]: pgmap v305: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:53.005 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:52 smithi055 ceph-mon[40587]: pgmap v305: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T21:35:53.015 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:53.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:53.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:53.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:53.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:53.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:53.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:53.649 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:35:53.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:53.747 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:53.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:53.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:53.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:53.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:53.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 45 16 78279 78263 108.675 69.25 0.00674276 0.00919452 2024-03-26T21:35:54.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:54.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:54.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:54.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:54.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:54.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:54.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:54.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:35:54.205+0000 7fb1ff446700 -1 mon.a@0(leader).osd e186 definitely_dead 0 2024-03-26T21:35:54.589 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:35:54.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:54.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:54.758 INFO:teuthology.orchestra.run.smithi055.stderr:marked out osd.4. 2024-03-26T21:35:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:35:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:35:54.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:54.820 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:54 smithi067 ceph-mon[33346]: pgmap v306: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 755 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T21:35:54.820 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:54 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1696258158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:54.820 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:54 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:54.820 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:54 smithi067 ceph-mon[33346]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-26T21:35:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:54.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:54.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:54.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:54.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:54.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 46 16 79255 79239 107.639 61 0.00388789 0.00924662 2024-03-26T21:35:55.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[29406]: pgmap v306: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 755 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T21:35:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1696258158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[29406]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-26T21:35:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[40587]: pgmap v306: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 755 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T21:35:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1696258158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-26T21:35:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:54 smithi055 ceph-mon[40587]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-26T21:35:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.085 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:55.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:55.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.212 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:55.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:55.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:55.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:55.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:55.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:55.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:55.593 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:55.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:55.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:55.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:35:55.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:55.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:55.907 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T21:35:55.908 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:55 smithi055 ceph-mon[40587]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T21:35:55.908 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T21:35:55.908 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:55 smithi055 ceph-mon[29406]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T21:35:55.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:55.974 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:55.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:55.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 47 16 79255 79239 105.349 0 - 0.00924662 2024-03-26T21:35:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-26T21:35:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:55 smithi067 ceph-mon[33346]: osdmap e187: 8 total, 8 up, 7 in 2024-03-26T21:35:56.008 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:35:56.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:56.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:56.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:56.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:56.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:56.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:56.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:56.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:56.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:56.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:56.731 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:56.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:56.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:56.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:56.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:56.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 48 16 79312 79296 103.228 1.78125 0.00519982 0.0094868 2024-03-26T21:35:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:56 smithi067 ceph-mon[33346]: pgmap v308: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T21:35:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:56 smithi067 ceph-mon[33346]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T21:35:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:56 smithi067 ceph-mon[33346]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T21:35:56.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:35:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:35:56.999 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[40587]: pgmap v308: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T21:35:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[40587]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T21:35:57.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[40587]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T21:35:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[29406]: pgmap v308: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T21:35:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[29406]: osdmap e188: 8 total, 8 up, 7 in 2024-03-26T21:35:57.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:56 smithi055 ceph-mon[29406]: osdmap e189: 8 total, 8 up, 7 in 2024-03-26T21:35:57.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:57.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:57.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:57.418 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:57.520 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.562 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:57.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:57.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.702 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:57.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:57.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:57.810 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:35:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:57.855 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[40587]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 186'5049 2024-03-26T21:35:57.855 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[40587]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 186'4969 2024-03-26T21:35:57.855 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:57.855 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4074096765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:57.856 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:35:57.856 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[29406]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 186'5049 2024-03-26T21:35:57.856 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[29406]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 186'4969 2024-03-26T21:35:57.856 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:57.856 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4074096765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:57.898 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:35:57.938 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:35:57.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:57.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:57.977 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:35:57.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 49 16 80240 80224 102.305 58 0.00944361 0.00976941 2024-03-26T21:35:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:57 smithi067 ceph-mon[33346]: 8.2 starting backfill to osd.3 from (0'0,0'0] MAX to 186'5049 2024-03-26T21:35:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:57 smithi067 ceph-mon[33346]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 186'4969 2024-03-26T21:35:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:35:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4074096765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:35:58.025 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:35:58.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:58.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:35:58.112 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:58.125 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:35:58.136 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:35:58.137 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:35:58.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:58.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:35:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(8.5 189/188 [PushReplyOp(8:af3dfec7:::benchmark_data_smithi055_110126_object633:head)])", 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:58.383782+0000", 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.015690454999999999, 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015724331000000001, 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:58.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383782+0000", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383782+0000", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383783+0000", 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4600000000000003e-07 2024-03-26T21:35:58.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383789+0000", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5089999999999999e-06 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383804+0000", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4442e-05 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.383814+0000", 2024-03-26T21:35:58.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9799999999999993e-06 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.392721+0000", 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089073510000000009 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:80594 8.5 8.89d7e315 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:58.390095+0000", 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0093770980000000004, 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094710609999999994, 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:58.396 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 80594 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390095+0000", 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390095+0000", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390097+0000", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.59e-06 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390136+0000", 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9722999999999999e-05 2024-03-26T21:35:58.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390153+0000", 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6750999999999999e-05 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390162+0000", 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4489999999999999e-06 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.399 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:80597 8.5 8.6db084b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:58.390176+0000", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0092959319999999998, 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094449930000000005, 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 80597 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:58.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390176+0000", 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390176+0000", 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:58.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390177+0000", 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2200000000000004e-07 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390205+0000", 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8405999999999999e-05 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390212+0000", 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5830000000000003e-06 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.390217+0000", 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5090000000000001e-06 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:80584 8.7 e189/173 8:e2a061a6:::benchmark_data_smithi055_110126_object80583:head v 189'5087, mlcod=189'5085)", 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:58.391170+0000", 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0083022329999999991, 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085054910000000004, 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:58.403 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391170+0000", 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391170+0000", 2024-03-26T21:35:58.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391171+0000", 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5300000000000004e-07 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391245+0000", 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3924999999999994e-05 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391253+0000", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7090000000000008e-06 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391262+0000", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0469999999999999e-06 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391332+0000", 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9332999999999995e-05 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.391350+0000", 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476999999999999e-05 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:80585 8.4 e189/188 8:2e35b596:::benchmark_data_smithi055_110126_object80584:head v 189'4966, mlcod=0'0)", 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:35:58.394145+0000", 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0053268370000000001, 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055877310000000003, 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:35:58.408 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394145+0000", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394145+0000", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394146+0000", 2024-03-26T21:35:58.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.108e-06 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394170+0000", 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3781000000000002e-05 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394195+0000", 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5224000000000001e-05 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:35:58.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394204+0000", 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1790000000000008e-06 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394310+0000", 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105888 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:35:58.394335+0000", 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5207000000000001e-05 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:35:58.412 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:35:58.412 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 5 2024-03-26T21:35:58.412 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:35:58.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:58.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:58.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:58.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:35:58.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:35:58.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:58.943 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:58.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:58.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 50 16 81173 81157 101.425 58.3125 0.0131581 0.00985355 2024-03-26T21:35:58.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:58 smithi055 ceph-mon[40587]: pgmap v311: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:35:58.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:35:58 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 4943/238428 objects degraded (2.073%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:35:58.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:58 smithi055 ceph-mon[29406]: pgmap v311: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:35:58.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:35:58 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 4943/238428 objects degraded (2.073%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:35:59.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:35:59.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:35:59.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:58 smithi067 ceph-mon[33346]: pgmap v311: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.30k op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:35:59.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:35:58 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 4943/238428 objects degraded (2.073%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:35:59.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:35:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.222 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:59.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:35:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:35:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:35:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:35:59.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:35:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:59.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:35:59.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:59.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:35:59.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:59.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:35:59.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:35:59.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:35:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:35:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:35:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:35:59.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:35:59.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:35:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:35:59.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:35:59.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 51 16 82091 82075 100.562 57.375 0.00877631 0.00993786 2024-03-26T21:36:00.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:00.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:00.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:00.220 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:00.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:00.357 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:00.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:00.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:00.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:00.768 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:00.831 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:36:00.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:00.890 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[29406]: pgmap v312: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 682 op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:36:00.890 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/179401253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:00.890 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:00.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[40587]: pgmap v312: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 682 op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:36:00.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/179401253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:00.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:00 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:00.907 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:00.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:00.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 52 16 82924 82908 99.6289 52.0625 0.0429272 0.010028 2024-03-26T21:36:01.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:01.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:01.077 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:36:01.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:00 smithi067 ceph-mon[33346]: pgmap v312: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 remapped+peering, 143 active+clean; 4.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 43 MiB/s wr, 682 op/s; 4943/238428 objects degraded (2.073%); 9980/238428 objects misplaced (4.186%); 640 KiB/s, 23 objects/s recovering 2024-03-26T21:36:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/179401253' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:00 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:01.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:01.309 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:36:01.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:36:01.309 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T21:36:01.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T21:36:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:01.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:01.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:01.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:01.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:01.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:01.740 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:01.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:01.881 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:01.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:01.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:01.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 53 16 83611 83595 98.5593 42.9375 0.0275521 0.0101396 2024-03-26T21:36:02.024 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:02.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:01 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:02.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:01 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:02.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:01 smithi067 ceph-mon[33346]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T21:36:02.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:02.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:02.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:02.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:02.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:02.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[29406]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T21:36:02.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:02.129 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:02.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:02.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:02.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[40587]: osdmap e190: 8 total, 8 up, 7 in 2024-03-26T21:36:02.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:02.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:02.164 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:02.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:02.304 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:02.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:02.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:02.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:02.576 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:02.591 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:36:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:36:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:36:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:36:02.592 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:36:02.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:02.707 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:02.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:02.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:02.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:02.967 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:02.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:02.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 54 16 84304 84288 97.5363 43.3125 0.0216567 0.0102354 2024-03-26T21:36:03.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:03.108 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:02 smithi067 ceph-mon[33346]: pgmap v314: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 817 op/s; 4943/242082 objects degraded (2.042%); 19518/242082 objects misplaced (8.063%); 3.3 MiB/s, 67 objects/s recovering 2024-03-26T21:36:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:02 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:36:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:02 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:36:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:03.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[29406]: pgmap v314: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 817 op/s; 4943/242082 objects degraded (2.042%); 19518/242082 objects misplaced (8.063%); 3.3 MiB/s, 67 objects/s recovering 2024-03-26T21:36:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:36:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:36:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[40587]: pgmap v314: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 817 op/s; 4943/242082 objects degraded (2.042%); 19518/242082 objects misplaced (8.063%); 3.3 MiB/s, 67 objects/s recovering 2024-03-26T21:36:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:36:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:02 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:36:03.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:36:03.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:03.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:03.502 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.541 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:03.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:03.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:03.668 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:03.670 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:03.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:03.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:03.939 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:03.977 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:03.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:03.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 55 16 84961 84945 96.5095 41.0625 0.0186408 0.0103514 2024-03-26T21:36:04.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:04.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:04.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:03 smithi067 ceph-mon[33346]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T21:36:04.251 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:04.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:03 smithi055 ceph-mon[29406]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T21:36:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:03 smithi055 ceph-mon[40587]: osdmap e191: 8 total, 8 up, 7 in 2024-03-26T21:36:04.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.382 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:04.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:04.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:04.617 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:04.657 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:04.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:04.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:04.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:04.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:04.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:04.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 56 16 85746 85730 95.6622 49.0625 0.00800068 0.0104478 2024-03-26T21:36:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:04 smithi067 ceph-mon[33346]: pgmap v316: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 768 B/s rd, 52 MiB/s wr, 832 op/s; 23587/250827 objects misplaced (9.404%); 16 MiB/s, 255 objects/s recovering 2024-03-26T21:36:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:04 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4943/242082 objects degraded (2.042%), 1 pg degraded) 2024-03-26T21:36:05.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:05.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:05.217 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:04 smithi055 ceph-mon[29406]: pgmap v316: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 768 B/s rd, 52 MiB/s wr, 832 op/s; 23587/250827 objects misplaced (9.404%); 16 MiB/s, 255 objects/s recovering 2024-03-26T21:36:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:04 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4943/242082 objects degraded (2.042%), 1 pg degraded) 2024-03-26T21:36:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:04 smithi055 ceph-mon[40587]: pgmap v316: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 768 B/s rd, 52 MiB/s wr, 832 op/s; 23587/250827 objects misplaced (9.404%); 16 MiB/s, 255 objects/s recovering 2024-03-26T21:36:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:04 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4943/242082 objects degraded (2.042%), 1 pg degraded) 2024-03-26T21:36:05.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:05.354 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:05.456 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:05.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:05.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:05.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:05.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:05.753 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:05.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:05.892 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:05.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:05.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 57 15 86603 86588 94.9247 53.625 0.0100376 0.0105291 2024-03-26T21:36:05.993 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.034 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:06.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:06.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:06.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:06.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:06.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:06.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:06.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:06.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:06.576 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:06.678 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:06.715 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:06.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:06.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:06.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:06.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:06.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 58 16 87333 87317 94.0737 45.5625 0.0162678 0.0106246 2024-03-26T21:36:07.039 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:07.051 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:07.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.138 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:07.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:07.184 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:06 smithi055 ceph-mon[40587]: pgmap v317: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 23587/250827 objects misplaced (9.404%); 13 MiB/s, 202 objects/s recovering 2024-03-26T21:36:07.184 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:06 smithi055 ceph-mon[40587]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T21:36:07.184 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:06 smithi055 ceph-mon[29406]: pgmap v317: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 23587/250827 objects misplaced (9.404%); 13 MiB/s, 202 objects/s recovering 2024-03-26T21:36:07.184 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:06 smithi055 ceph-mon[29406]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T21:36:07.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:06 smithi067 ceph-mon[33346]: pgmap v317: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 23587/250827 objects misplaced (9.404%); 13 MiB/s, 202 objects/s recovering 2024-03-26T21:36:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:06 smithi067 ceph-mon[33346]: osdmap e192: 8 total, 8 up, 7 in 2024-03-26T21:36:07.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:07.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:07.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:07.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:07.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:07.564 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:07.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:07.698 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:07.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.834 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:07.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:07.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:07.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:07.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:07.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 59 16 88250 88234 93.4506 57.3125 0.026222 0.0106953 2024-03-26T21:36:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:08.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:08.104 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:08.169 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:08.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:08.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:08.375 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1123 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94208173341440 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 337.64655738599998, 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0115813600000001, 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:36:08.376 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1123 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732483+0000", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2589999999999999e-06 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732488+0000", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7539999999999997e-06 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732499+0000", 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1243000000000001e-05 2024-03-26T21:36:08.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732508+0000", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0040000000000005e-06 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732567+0000", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8811000000000003e-05 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732573+0000", 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2330000000000003e-06 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736258+0000", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036856459999999999 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736267+0000", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7460000000000006e-06 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739066+0000", 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.002799072 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739206+0000", 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013932300000000001 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.744063+0000", 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004857269 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.381 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1124 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94208178013312 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 337.64639515300001, 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.011248098, 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1124 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4099999999999998e-07 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732648+0000", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.771e-06 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732657+0000", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9639999999999992e-06 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732679+0000", 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1985999999999999e-05 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732893+0000", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021387399999999999 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732903+0000", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9499999999999996e-06 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738098+0000", 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005195499 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738271+0000", 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172795 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.743892+0000", 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005620618 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118015 7.c 7:3ece5431:::benchmark_data_smithi055_81491_object118014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:36:08.386 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 226.44255892800001, 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12752479, 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118015 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936480+0000", 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936489+0000", 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6119999999999999e-06 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936522+0000", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3831999999999997e-05 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936531+0000", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0749999999999998e-06 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936537+0000", 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7549999999999997e-06 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936603+0000", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6093999999999994e-05 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936788+0000", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018466199999999999 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936962+0000", 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017412300000000001 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942702+0000", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057399269999999997 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942741+0000", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9239999999999997e-05 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.063918+0000", 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121176487 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.063965+0000", 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7635e-05 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064005+0000", 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9347999999999997e-05 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.392 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:118016 7.c 7:32014175:::benchmark_data_smithi055_81491_object118015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 226.44243024599999, 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12750198700000001, 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 118016 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936609+0000", 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.68e-07 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936637+0000", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7277999999999999e-05 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936644+0000", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-06 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.936651+0000", 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8950000000000001e-06 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940183+0000", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035317270000000001 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940344+0000", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016089000000000001 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.940539+0000", 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019523199999999999 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942824+0000", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022846260000000001 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:21.942871+0000", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7315999999999999e-05 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064045+0000", 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121174358 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064082+0000", 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6288000000000002e-05 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:22.064111+0000", 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9005000000000001e-05 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:196604 7.c 7:3d274679:::benchmark_data_smithi055_81491_object77117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:36:08.398 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:47.605950+0000", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 140.773088839, 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.123939074, 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 196604 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605950+0000", 2024-03-26T21:36:08.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605950+0000", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605951+0000", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0240000000000001e-06 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605955+0000", 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.25e-06 2024-03-26T21:36:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605965+0000", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3009999999999995e-06 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.605972+0000", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5650000000000004e-06 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.606017+0000", 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4821000000000003e-05 2024-03-26T21:36:08.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.606118+0000", 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100554 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,7", 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.606199+0000", 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1738999999999998e-05 2024-03-26T21:36:08.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.608989+0000", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027896800000000001 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.609071+0000", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1791000000000003e-05 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.729819+0000", 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.120748011 2024-03-26T21:36:08.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.729854+0000", 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4696000000000001e-05 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:47.729889+0000", 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5642000000000002e-05 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.405 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197497 7.d 7:ba42a995:::benchmark_data_smithi055_81491_object78010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.64182768800001, 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.166716791, 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197497 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737213+0000", 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3599999999999999e-06 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737222+0000", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.842e-06 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737236+0000", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3692e-05 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737244+0000", 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3989999999999998e-06 2024-03-26T21:36:08.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737302+0000", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8579000000000003e-05 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737433+0000", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013079199999999999 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,1", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737563+0000", 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000129846 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.740324+0000", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027609940000000001 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.741152+0000", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082808100000000004 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903860+0000", 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.162708202 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903889+0000", 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9422999999999999e-05 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903928+0000", 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8581000000000002e-05 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.411 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197503 7.7 7:e387ae8a:::benchmark_data_smithi055_81491_object78016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.63540017400001, 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15943073799999999, 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197503 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:36:08.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743640+0000", 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.23e-07 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743644+0000", 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4279999999999998e-06 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743653+0000", 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4429999999999992e-06 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743661+0000", 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6090000000000003e-06 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743706+0000", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4637e-05 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743794+0000", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8764999999999995e-05 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743873+0000", 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8942000000000004e-05 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.746188+0000", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023149080000000001 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.895858+0000", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14966966300000001 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.902961+0000", 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071028239999999998 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903017+0000", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6657000000000003e-05 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903069+0000", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2039000000000001e-05 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:83817 8.5 8:a1d2a373:::benchmark_data_smithi055_110126_object83816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.248574+0000", 2024-03-26T21:36:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.1304645349999998, 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.124832419, 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 83817 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248574+0000", 2024-03-26T21:36:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248574+0000", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248576+0000", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.407e-06 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248636+0000", 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0297999999999998e-05 2024-03-26T21:36:08.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248651+0000", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4564e-05 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.248663+0000", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2709e-05 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.260306+0000", 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011642635 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.260479+0000", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017260600000000001 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.263753+0000", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00327415 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.274267+0000", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010514651999999999 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.277330+0000", 2024-03-26T21:36:08.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030630840000000002 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.369557+0000", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.092227003000000002 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.369617+0000", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9506000000000003e-05 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.373407+0000", 2024-03-26T21:36:08.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037898049999999998 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84024 8.5 8:a9a6864a:::benchmark_data_smithi055_110126_object84023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8475442900000001, 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18444987099999999, 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.423 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84024 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531496+0000", 2024-03-26T21:36:08.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.73e-06 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531543+0000", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6681999999999998e-05 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531556+0000", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3032e-05 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531564+0000", 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1289999999999996e-06 2024-03-26T21:36:08.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555104+0000", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023539966999999998 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555293+0000", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018880700000000001 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555499+0000", 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020556200000000001 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564809+0000", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093100309999999999 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567429+0000", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002620645 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712266+0000", 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14483680500000001 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712311+0000", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4746000000000002e-05 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.715945+0000", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036337349999999999 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84035 8.5 8:a52a9b25:::benchmark_data_smithi055_110126_object84034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:36:08.428 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8347530509999999, 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.17997925100000001, 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84035 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:36:08.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544287+0000", 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4079999999999999e-06 2024-03-26T21:36:08.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544397+0000", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010953 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544421+0000", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4443999999999998e-05 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544429+0000", 2024-03-26T21:36:08.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6680000000000001e-06 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558408+0000", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013978945 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558584+0000", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017627100000000001 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558791+0000", 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020727899999999999 2024-03-26T21:36:08.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567383+0000", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085917289999999993 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567490+0000", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010709000000000001 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.720857+0000", 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15336719900000001 2024-03-26T21:36:08.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.720902+0000", 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4885000000000001e-05 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724265+0000", 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033628030000000001 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84036 8.5 8:a733162b:::benchmark_data_smithi055_110126_object84035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8345928269999998, 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18335849000000001, 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84036 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544447+0000", 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.78e-07 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544508+0000", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0568e-05 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544530+0000", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2268000000000001e-05 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544536+0000", 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2829999999999998e-06 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564845+0000", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020308641999999998 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565017+0000", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017215199999999999 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565220+0000", 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020294399999999999 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576111+0000", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010890982 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576274+0000", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000162819 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724342+0000", 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14806820500000001 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724408+0000", 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6094999999999997e-05 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727805+0000", 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033966539999999998 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.440 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84038 8.5 8:a3a8abbc:::benchmark_data_smithi055_110126_object84037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:36:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8344941089999995, 2024-03-26T21:36:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18689033499999999, 2024-03-26T21:36:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:08.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84038 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.06e-07 2024-03-26T21:36:08.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544585+0000", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9360000000000003e-05 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544592+0000", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-06 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544598+0000", 2024-03-26T21:36:08.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5310000000000001e-06 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567532+0000", 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022934059 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567708+0000", 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017579499999999999 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567936+0000", 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000228173 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576172+0000", 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082357030000000005 2024-03-26T21:36:08.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576281+0000", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109747 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727868+0000", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.151586689 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727982+0000", 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011343700000000001 2024-03-26T21:36:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731435+0000", 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003453631 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84039 8.5 8:a8845c01:::benchmark_data_smithi055_110126_object84038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8344323170000001, 2024-03-26T21:36:08.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.190558218, 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84039 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4400000000000001e-07 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544631+0000", 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.442e-05 2024-03-26T21:36:08.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544638+0000", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3790000000000003e-06 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544643+0000", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7160000000000002e-06 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570111+0000", 2024-03-26T21:36:08.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025468094 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570292+0000", 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018180200000000001 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570501+0000", 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208727 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576257+0000", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057555239999999997 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584951+0000", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086946679999999991 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731584+0000", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.146632868 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731658+0000", 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3287999999999994e-05 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735165+0000", 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035072879999999999 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84048 8.5 8:a98c3a1d:::benchmark_data_smithi055_110126_object84047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.822967212, 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.182857732, 2024-03-26T21:36:08.456 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84048 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556073+0000", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.584e-06 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556178+0000", 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104714 2024-03-26T21:36:08.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556189+0000", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0582e-05 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556196+0000", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8259999999999993e-06 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573081+0000", 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016884952000000002 2024-03-26T21:36:08.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573261+0000", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017950600000000001 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573469+0000", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020786200000000001 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584956+0000", 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011487264000000001 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585011+0000", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4629000000000002e-05 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735223+0000", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.150211927 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735322+0000", 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9017000000000004e-05 2024-03-26T21:36:08.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.738929+0000", 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036078690000000001 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84053 8.5 8:a46639cb:::benchmark_data_smithi055_110126_object84052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8150663179999995, 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.17847104499999999, 2024-03-26T21:36:08.462 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84053 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563974+0000", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4449999999999999e-06 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564042+0000", 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8171999999999997e-05 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564054+0000", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1537e-05 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564062+0000", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2740000000000007e-06 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576328+0000", 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012265728 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576475+0000", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014767799999999999 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576658+0000", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018290400000000001 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585057+0000", 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0083985689999999998 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585066+0000", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9139999999999997e-06 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.738983+0000", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15391669699999999 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.739084+0000", 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010157000000000001 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.742444+0000", 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003359557 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84738 8.5 8:ad591b1d:::benchmark_data_smithi055_110126_object84737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.777983088, 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.165289678, 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84738 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601057+0000", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.437e-06 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601116+0000", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9123000000000003e-05 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601126+0000", 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3039999999999994e-06 2024-03-26T21:36:08.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601132+0000", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5939999999999999e-06 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.611839+0000", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010707003999999999 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.611997+0000", 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157935 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.612193+0000", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196219 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.628571+0000", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016377571 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633071+0000", 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045002790000000003 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.763063+0000", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12999149400000001 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.763143+0000", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0453999999999999e-05 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766346+0000", 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032022639999999998 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.473 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84759 8.5 8:ab1dad65:::benchmark_data_smithi055_110126_object84758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7601245399999996, 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14978145100000001, 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84759 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618916+0000", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7349999999999999e-06 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618958+0000", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1878000000000001e-05 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618969+0000", 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.084e-05 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618977+0000", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8890000000000002e-06 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625426+0000", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006449092 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625618+0000", 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019185299999999999 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625829+0000", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210965 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633120+0000", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072909649999999999 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640696+0000", 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075759269999999997 2024-03-26T21:36:08.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766432+0000", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.125736666 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766478+0000", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5377999999999998e-05 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768696+0000", 2024-03-26T21:36:08.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022182629999999998 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84763 8.5 8:ae380329:::benchmark_data_smithi055_110126_object84762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7556124769999997, 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.147994242, 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84763 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623435+0000", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0280000000000006e-06 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623469+0000", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3723000000000002e-05 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623484+0000", 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4858000000000001e-05 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623492+0000", 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6879999999999999e-06 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633153+0000", 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096608590000000008 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633302+0000", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149326 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633480+0000", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017833700000000001 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640675+0000", 2024-03-26T21:36:08.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071944629999999999 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640704+0000", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768765+0000", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12806050699999999 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768831+0000", 2024-03-26T21:36:08.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6161999999999994e-05 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.771421+0000", 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025899830000000001 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84768 8.5 8:afce3940:::benchmark_data_smithi055_110126_object84767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:36:08.486 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7492226799999999, 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.166609485, 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84768 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:36:08.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629818+0000", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7179999999999999e-06 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629872+0000", 2024-03-26T21:36:08.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4032e-05 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629886+0000", 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.431e-05 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629894+0000", 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0139999999999998e-06 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637716+0000", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078216980000000002 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637805+0000", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9189000000000003e-05 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637951+0000", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014564800000000001 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640755+0000", 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028039010000000001 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.641376+0000", 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062082399999999996 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.792216+0000", 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.150839943 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.792277+0000", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.19e-05 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.796426+0000", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041483079999999999 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:86639 8.5 8:a036fec7:::benchmark_data_smithi055_110126_object86638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.349466429, 2024-03-26T21:36:08.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.236906851, 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 86639 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029574+0000", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.271e-06 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029628+0000", 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4159000000000001e-05 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029639+0000", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0638e-05 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029646+0000", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1629999999999999e-06 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.036918+0000", 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072720809999999997 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037144+0000", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022611500000000001 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037343+0000", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019870900000000001 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.040140+0000", 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00279715 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.042107+0000", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019675669999999999 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262797+0000", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.220689684 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262862+0000", 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5030000000000006e-05 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.266479+0000", 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036172840000000001 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:08.498 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:36:08.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:08.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:08.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:08.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:08.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:08.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:08.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:36:08.994697+0000 min lat: 0.0012867 max lat: 2.26773 avg lat: 0.0107586 2024-03-26T21:36:08.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:36:08.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 60 16 89217 89201 92.9003 60.4375 0.0344615 0.0107586 2024-03-26T21:36:08.996 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:08.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:09.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:36:09.047 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:08 smithi055 ceph-mon[40587]: pgmap v319: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 22356/261948 objects misplaced (8.535%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:36:09.047 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:08 smithi055 ceph-mon[29406]: pgmap v319: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 22356/261948 objects misplaced (8.535%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:36:09.047 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:09.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:09.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:09.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:08 smithi067 ceph-mon[33346]: pgmap v319: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 22356/261948 objects misplaced (8.535%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:36:09.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:09.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:09.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:36:09.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:09.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.589 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:09.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:09.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:09.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:09.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:09.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:09.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 61 16 90124 90108 92.3066 56.6875 0.027655 0.0108281 2024-03-26T21:36:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:10.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:10.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:10.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:10.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:10.389 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:10.431 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:10.533 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.575 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:10.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:10.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:10.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:10.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:10.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:10.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 62 15 90960 90945 91.6616 52.3125 0.0338336 0.0109041 2024-03-26T21:36:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:10.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:11.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:11.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:10 smithi067 ceph-mon[33346]: pgmap v320: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 828 op/s; 22356/261948 objects misplaced (8.535%); 23 MiB/s, 373 objects/s recovering 2024-03-26T21:36:11.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:11.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:10 smithi055 ceph-mon[29406]: pgmap v320: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 828 op/s; 22356/261948 objects misplaced (8.535%); 23 MiB/s, 373 objects/s recovering 2024-03-26T21:36:11.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:10 smithi055 ceph-mon[40587]: pgmap v320: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 828 op/s; 22356/261948 objects misplaced (8.535%); 23 MiB/s, 373 objects/s recovering 2024-03-26T21:36:11.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:11.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:11.518 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:11.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:11.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:11.690 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:11.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:11.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:11.803 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:91607 8.6 e192/173 8:6a99dc6d:::benchmark_data_smithi055_110126_object91606:head v 192'5709, mlcod=192'5707)", 2024-03-26T21:36:11.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.820525+0000", 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.018628201, 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018658953999999998, 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820525+0000", 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:11.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820525+0000", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820525+0000", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5999999999999999e-07 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820557+0000", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1930000000000001e-05 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:11.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820565+0000", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7449999999999995e-06 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820574+0000", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6300000000000004e-06 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.826568+0000", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005994821 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:11.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.826592+0000", 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3802e-05 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:91608 8.e e192/188 8:76f7d0c7:::benchmark_data_smithi055_110126_object91607:head v 192'5704, mlcod=0'0)", 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.820613+0000", 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.018540338, 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018636201000000002, 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:11.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820613+0000", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820613+0000", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820613+0000", 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1699999999999999e-07 2024-03-26T21:36:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820643+0000", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9680999999999999e-05 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820649+0000", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7900000000000002e-06 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.820655+0000", 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8769999999999998e-06 2024-03-26T21:36:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.830011+0000", 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093554429999999997 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.830044+0000", 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3346000000000002e-05 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:91619 8.e e192/188 8:7682fbc1:::benchmark_data_smithi055_110126_object91618:head v 192'5705, mlcod=0'0)", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.821824+0000", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.017328633, 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017484158999999999, 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821824+0000", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821824+0000", 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821825+0000", 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7e-07 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821867+0000", 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2298000000000001e-05 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821877+0000", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0342e-05 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.821883+0000", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8170000000000004e-06 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835957+0000", 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014073337999999999 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835987+0000", 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9865e-05 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:91630 8.6 e192/173 8:6f82907b:::benchmark_data_smithi055_110126_object91629:head v 192'5710, mlcod=192'5707)", 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.835478+0000", 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0036749339999999999, 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038886599999999999, 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835478+0000", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835478+0000", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835479+0000", 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8599999999999997e-07 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835508+0000", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835521+0000", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3178e-05 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835529+0000", 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8890000000000002e-06 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835872+0000", 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034281900000000001 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.835892+0000", 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0806e-05 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15096.0:91633 8.e e192/188)", 2024-03-26T21:36:11.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.837172+0000", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.001981273, 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022501700000000001, 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837172+0000", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837172+0000", 2024-03-26T21:36:11.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837173+0000", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9299999999999997e-07 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837284+0000", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011178100000000001 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837299+0000", 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4280999999999999e-05 2024-03-26T21:36:11.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.837307+0000", 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9419999999999995e-06 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 5 2024-03-26T21:36:11.850 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:36:11.851 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:36:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.852 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:11.868 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:36:11.925 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:36:11.950 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:36:11.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:11.974 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:36:11.978 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:11.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:11.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 63 16 91667 91651 90.907 44.125 0.0696013 0.0109816 2024-03-26T21:36:12.011 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:36:12.054 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:36:12.062 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:36:12.062 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:36:12.062 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:36:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:36:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:36:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:36:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:36:12.063 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:36:12.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.119 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:12.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:12.254 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:12.356 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:12.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:12.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:12.523 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:12.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:12.668 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:12.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:12.793 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:12.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:12 smithi055 ceph-mon[40587]: pgmap v321: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 527 op/s; 22202/264375 objects misplaced (8.398%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:36:12.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/535771438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:12 smithi055 ceph-mon[29406]: pgmap v321: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 527 op/s; 22202/264375 objects misplaced (8.398%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:36:12.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/535771438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:12.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:12.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:12.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:12.946 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:12 smithi067 ceph-mon[33346]: pgmap v321: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 33 MiB/s wr, 527 op/s; 22202/264375 objects misplaced (8.398%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:36:12.946 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/535771438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:12.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 64 16 92329 92313 90.1331 41.375 0.00873392 0.0110887 2024-03-26T21:36:12.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:36:13.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:13.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:13.183 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:13.221 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:13.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:13.366 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:13.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.502 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:13.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:13.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:13.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:13.763 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:13.801 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:13 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3442187323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:36:13.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:13.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:13.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 65 16 93014 92998 89.405 42.8125 0.0418065 0.0111669 2024-03-26T21:36:14.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:14.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:14.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3442187323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:36:14.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:13 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3442187323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:36:14.126 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:36:14.126 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:36:14.126 INFO:teuthology.orchestra.run.smithi055.stdout: "epoch": 192, 2024-03-26T21:36:14.126 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:36:14.126 INFO:teuthology.orchestra.run.smithi055.stdout: "created": "2024-03-26T21:22:56.673257+0000", 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "modified": "2024-03-26T21:36:06.630752+0000", 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "last_up_change": "2024-03-26T21:32:54.443212+0000", 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "last_in_change": "2024-03-26T21:35:54.747565+0000", 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_num": 5806080, 2024-03-26T21:36:14.127 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_set": [ 2024-03-26T21:36:14.128 INFO:teuthology.orchestra.run.smithi055.stdout: "nodeep-scrub", 2024-03-26T21:36:14.128 INFO:teuthology.orchestra.run.smithi055.stdout: "noscrub", 2024-03-26T21:36:14.128 INFO:teuthology.orchestra.run.smithi055.stdout: "pglog_hardlimit", 2024-03-26T21:36:14.128 INFO:teuthology.orchestra.run.smithi055.stdout: "purged_snapdirs", 2024-03-26T21:36:14.128 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_deletes", 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: "sortbitwise" 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_version": 19, 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T21:36:14.129 INFO:teuthology.orchestra.run.smithi055.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_snapshot": "", 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_max": 8, 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "max_osd": 8, 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "require_min_compat_client": "luminous", 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "min_compat_client": "luminous", 2024-03-26T21:36:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: "require_osd_release": "squid", 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: "allow_crimson": false, 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: "pools": [ 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 1, 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".mgr", 2024-03-26T21:36:14.131 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:26:13.884991+0000", 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.132 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 1, 2024-03-26T21:36:14.133 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 1, 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 1, 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 1, 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 1, 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.134 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.135 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "112", 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.136 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.137 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.138 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.139 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.140 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.141 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_max": 32, 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_min": 1 2024-03-26T21:36:14.142 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": {} 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.143 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 7.8899998664855957, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 7.8899998664855957, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 3, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 3, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.144 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 2, 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "datapool", 2024-03-26T21:36:14.145 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.142617+0000", 2024-03-26T21:36:14.146 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 8193, 2024-03-26T21:36:14.146 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T21:36:14.146 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.146 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.146 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.147 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 3, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 3, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 3, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 3, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 3, 2024-03-26T21:36:14.148 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.149 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.150 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.150 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "71", 2024-03-26T21:36:14.150 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.150 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.150 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 2, 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 71, 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.151 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.152 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.153 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.154 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.155 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.156 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.156 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.156 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.156 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.156 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd": {} 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.157 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.6500000953674316, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.6500000953674316, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.158 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 3, 2024-03-26T21:36:14.159 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".rgw.root", 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.591599+0000", 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.160 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.161 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:36:14.162 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.163 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "68", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.164 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.165 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.166 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.7400000095367432, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.7400000095367432, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 4, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.log", 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:34.904126+0000", 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.168 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "70", 2024-03-26T21:36:14.169 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.170 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.171 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:36:14.172 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.4900000095367432, 2024-03-26T21:36:14.173 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.4900000095367432, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 5, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.control", 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:36.852373+0000", 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.174 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:36:14.175 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "72", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.176 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.177 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.178 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.7400000095367432, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.7400000095367432, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.179 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 6, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.meta", 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:38.937273+0000", 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.180 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.181 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "74", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.182 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.183 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_bias": 4 2024-03-26T21:36:14.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.9900000095367432, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.9900000095367432, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 8, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "unique_pool_1", 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:34:56.999064+0000", 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:36:14.186 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 16, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 16, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 16, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 16, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 16, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:36:14.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "176", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:36:14.188 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:36:14.189 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "rados": {} 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:36:14.190 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.9900000095367432, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.9900000095367432, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:36:14.191 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae", 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 8, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 188, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.192 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6801", 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6802", 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.193 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6804", 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6803", 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.194 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6801/317381212", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6802/317381212", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6804/317381212", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6803/317381212", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db", 2024-03-26T21:36:14.195 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 13, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 187, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6805", 2024-03-26T21:36:14.196 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6806", 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.197 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6808", 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6807", 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.198 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6805/3733554168", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6806/3733554168", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6808/3733554168", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6807/3733554168", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b", 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.199 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 18, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 187, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6809", 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:36:14.200 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6810", 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.201 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6812", 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6811", 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6809/3059242923", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6810/3059242923", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6812/3059242923", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6811/3059242923", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b", 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.203 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 26, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 188, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6813", 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.204 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6814", 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6816", 2024-03-26T21:36:14.205 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6815", 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6813/3851321468", 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6814/3851321468", 2024-03-26T21:36:14.206 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6816/3851321468", 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6815/3851321468", 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598", 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 0, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 0, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 31, 2024-03-26T21:36:14.207 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 92, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 133, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 173, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 93, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6800", 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.208 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6801", 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6803", 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:36:14.209 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6802", 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6800/254752278", 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6801/254752278", 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6803/254752278", 2024-03-26T21:36:14.210 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6802/254752278", 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213", 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.211 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 41, 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 188, 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6804", 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6805", 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6807", 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.213 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6806", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6804/278711358", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6805/278711358", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6807/278711358", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6806/278711358", 2024-03-26T21:36:14.214 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9", 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.215 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 50, 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 188, 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6808", 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6809", 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6811", 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6810", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6808/3926251639", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6809/3926251639", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6811/3926251639", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6810/3926251639", 2024-03-26T21:36:14.218 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55", 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 59, 2024-03-26T21:36:14.219 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 187, 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6812", 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6813", 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6815", 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6814", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 2981645260 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6812/2981645260", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6813/2981645260", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6815/2981645260", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6814/2981645260", 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:36:14.222 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_xinfo": [ 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:07.744892+0000", 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:36:02.334700+0000", 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.224 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:20.324272+0000", 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:27.151111+0000", 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 65536, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:27.376024+0000", 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 92 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.226 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:37.372411+0000", 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:49.638606+0000", 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:35:48.732626+0000", 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap": [ 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "5.1a", 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.228 INFO:teuthology.orchestra.run.smithi055.stdout: 5, 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: 6, 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: 0 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_items": [ 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.4", 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.6", 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.17", 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.230 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 2 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_primaries": [], 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_temp": [ 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "8.2", 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: 5, 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: 4 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "8.4", 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: 5, 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: 2 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "8.5", 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: 6, 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: 7 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "8.b", 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: 3, 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: 1, 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: 4 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "8.e", 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: 2 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.233 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_temp": [], 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklist": { 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2649740776": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/4141219490": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/147833426": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2854772543": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/2804789175": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2199694469": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1614234257": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2834138692": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1740219825": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/268664696": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/536717527": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:36:14.234 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/966571165": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2757111431": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/559658018": "2024-03-27T21:23:57.787077+0000" 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "range_blocklist": {}, 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profiles": { 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "default": { 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "crush-failure-domain": "osd", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "k": "2", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "m": "1", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin": "jerasure", 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "technique": "reed_sol_van" 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps_queue": [], 2024-03-26T21:36:14.235 INFO:teuthology.orchestra.run.smithi055.stdout: "new_removed_snaps": [], 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "new_purged_snaps": [], 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_node_flags": {}, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "device_class_flags": {}, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode": { 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_enabled": false, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_bucket_count": 0, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "degraded_stretch_mode": 0, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "recovering_stretch_mode": 0, 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_bucket": 0 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:36:14.236 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:36:14.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.260 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:14.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:14.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:14.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:14.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:14.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:14.814 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:14 smithi067 ceph-mon[33346]: pgmap v322: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:14.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:14.955 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:14.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:14.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 66 16 93844 93828 88.8363 51.875 0.00873031 0.0112512 2024-03-26T21:36:15.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:15.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:14 smithi055 ceph-mon[29406]: pgmap v322: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:14 smithi055 ceph-mon[40587]: pgmap v322: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:15.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:15.237 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:15.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:15.385 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:15.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:15.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:15.578 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:36:15.579 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:36:15.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:15.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:15.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:15.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:15.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:15.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 67 16 94810 94794 88.4115 60.375 0.0292101 0.0113052 2024-03-26T21:36:16.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:16.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:16.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:16.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:16.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:16.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:36:16.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:16.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:16.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:16.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:16.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:16.770 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:16.808 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:16.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:16.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:16.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:16.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 68 16 95760 95744 87.9845 59.375 0.0107419 0.0113604 2024-03-26T21:36:16.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:16 smithi067 ceph-mon[33346]: pgmap v323: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:16.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:16.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:16.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:17.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:17.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[40587]: pgmap v323: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:17.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:17.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[29406]: pgmap v323: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s; 20939/275253 objects misplaced (7.607%); 22 MiB/s, 344 objects/s recovering 2024-03-26T21:36:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:17.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.195 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:17.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:17.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:17.332 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:17.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:17.472 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:17.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:17.612 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:17.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:17.732 INFO:tasks.thrashosds.thrasher:j is {'epoch': 192, 'fsid': 'e5f52b02-ebb6-11ee-95d0-87774f69a715', 'created': '2024-03-26T21:22:56.673257+0000', 'modified': '2024-03-26T21:36:06.630752+0000', 'last_up_change': '2024-03-26T21:32:54.443212+0000', 'last_in_change': '2024-03-26T21:35:54.747565+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T21:28:33.142617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T21:28:33.591599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T21:28:34.904126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T21:28:36.852373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T21:28:38.937273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-26T21:34:56.999064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4f487071-c151-406b-b25b-3135cb3c63ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6801', 'nonce': 317381212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6802', 'nonce': 317381212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6804', 'nonce': 317381212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6803', 'nonce': 317381212}]}, 'public_addr': '172.21.15.55:6801/317381212', 'cluster_addr': '172.21.15.55:6802/317381212', 'heartbeat_back_addr': '172.21.15.55:6804/317381212', 'heartbeat_front_addr': '172.21.15.55:6803/317381212', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6805', 'nonce': 3733554168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6806', 'nonce': 3733554168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6808', 'nonce': 3733554168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6807', 'nonce': 3733554168}]}, 'public_addr': '172.21.15.55:6805/3733554168', 'cluster_addr': '172.21.15.55:6806/3733554168', 'heartbeat_back_addr': '172.21.15.55:6808/3733554168', 'heartbeat_front_addr': '172.21.15.55:6807/3733554168', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6809', 'nonce': 3059242923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6810', 'nonce': 3059242923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6812', 'nonce': 3059242923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6811', 'nonce': 3059242923}]}, 'public_addr': '172.21.15.55:6809/3059242923', 'cluster_addr': '172.21.15.55:6810/3059242923', 'heartbeat_back_addr': '172.21.15.55:6812/3059242923', 'heartbeat_front_addr': '172.21.15.55:6811/3059242923', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '30637a8d-5c1b-4ede-8214-a4595c875d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6813', 'nonce': 3851321468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6814', 'nonce': 3851321468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6816', 'nonce': 3851321468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6815', 'nonce': 3851321468}]}, 'public_addr': '172.21.15.55:6813/3851321468', 'cluster_addr': '172.21.15.55:6814/3851321468', 'heartbeat_back_addr': '172.21.15.55:6816/3851321468', 'heartbeat_front_addr': '172.21.15.55:6815/3851321468', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'cd2d68be-4b10-4242-826b-e085c2ce5598', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 92, 'up_from': 133, 'up_thru': 173, 'down_at': 93, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 254752278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6801', 'nonce': 254752278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 254752278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 254752278}]}, 'public_addr': '172.21.15.67:6800/254752278', 'cluster_addr': '172.21.15.67:6801/254752278', 'heartbeat_back_addr': '172.21.15.67:6803/254752278', 'heartbeat_front_addr': '172.21.15.67:6802/254752278', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 278711358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6805', 'nonce': 278711358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 278711358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 278711358}]}, 'public_addr': '172.21.15.67:6804/278711358', 'cluster_addr': '172.21.15.67:6805/278711358', 'heartbeat_back_addr': '172.21.15.67:6807/278711358', 'heartbeat_front_addr': '172.21.15.67:6806/278711358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd47c79f3-96bf-46b6-a7f3-a315446d04b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 188, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 3926251639}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 3926251639}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 3926251639}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 3926251639}]}, 'public_addr': '172.21.15.67:6808/3926251639', 'cluster_addr': '172.21.15.67:6809/3926251639', 'heartbeat_back_addr': '172.21.15.67:6811/3926251639', 'heartbeat_front_addr': '172.21.15.67:6810/3926251639', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f10b98b8-887d-4f83-8bc9-99f5c302dc55', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 2981645260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6813', 'nonce': 2981645260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 2981645260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2981645260}]}, 'public_addr': '172.21.15.67:6812/2981645260', 'cluster_addr': '172.21.15.67:6813/2981645260', 'heartbeat_back_addr': '172.21.15.67:6815/2981645260', 'heartbeat_front_addr': '172.21.15.67:6814/2981645260', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:07.744892+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:36:02.334700+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:20.324272+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:27.151111+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T21:35:27.376024+0000', 'dead_epoch': 92}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:37.372411+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:49.638606+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:35:48.732626+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.1a', 'osds': [5, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.2', 'osds': [0, 5, 4]}, {'pgid': '8.4', 'osds': [5, 2]}, {'pgid': '8.5', 'osds': [6, 7]}, {'pgid': '8.b', 'osds': [3, 1, 4]}, {'pgid': '8.e', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.55:0/2649740776': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/4141219490': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/147833426': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2854772543': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/2804789175': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/2199694469': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/1614234257': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/2834138692': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/1740219825': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/268664696': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:6800/536717527': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/966571165': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2757111431': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/559658018': '2024-03-27T21:23:57.787077+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T21:36:17.733 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.1a 2024-03-26T21:36:17.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd rm-pg-upmap 5.1a 2024-03-26T21:36:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:17.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:17.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:17.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:17.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:17.889 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:17.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:17.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 69 16 96721 96705 87.5798 60.0625 0.0135454 0.0114127 2024-03-26T21:36:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:17.990 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:18.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:18.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:18.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:18.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:18.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:36:18.405 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:18.491 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:18.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:18.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:18.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.779 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:18.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:18.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:18.922 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:18.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:18.987 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:18 smithi055 ceph-mon[40587]: pgmap v324: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 949 B/s rd, 64 MiB/s wr, 1.02k op/s; 19499/287283 objects misplaced (6.787%); 28 MiB/s, 451 objects/s recovering 2024-03-26T21:36:18.987 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:18 smithi055 ceph-mon[29406]: pgmap v324: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 949 B/s rd, 64 MiB/s wr, 1.02k op/s; 19499/287283 objects misplaced (6.787%); 28 MiB/s, 451 objects/s recovering 2024-03-26T21:36:18.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 70 16 97532 97516 87.0527 50.6875 0.00380463 0.0114772 2024-03-26T21:36:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:18 smithi067 ceph-mon[33346]: pgmap v324: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 949 B/s rd, 64 MiB/s wr, 1.02k op/s; 19499/287283 objects misplaced (6.787%); 28 MiB/s, 451 objects/s recovering 2024-03-26T21:36:19.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:19.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:19.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.212 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:19.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:19.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:19.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:19.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:19.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:19.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:19.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:19.614 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:19.715 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:19.761 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:19.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:19.897 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:19.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:19.932 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:19.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 71 16 98246 98230 86.4552 44.625 0.0174837 0.0115604 2024-03-26T21:36:19.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:20.005 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:20.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:20.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:20.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:20.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:20.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:20.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:20.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:20.559 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:20.595 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:20.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:20.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:20.745 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:36:20.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:20.873 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:20.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:20.883 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:20 smithi067 ceph-mon[33346]: pgmap v325: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 704 op/s; 19499/287283 objects misplaced (6.787%); 19 MiB/s, 296 objects/s recovering 2024-03-26T21:36:20.884 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/212282861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:20.884 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:20 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:20.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:20.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 72 16 99144 99128 86.0335 56.125 0.00452551 0.0116176 2024-03-26T21:36:21.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[40587]: pgmap v325: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 704 op/s; 19499/287283 objects misplaced (6.787%); 19 MiB/s, 296 objects/s recovering 2024-03-26T21:36:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/212282861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[29406]: pgmap v325: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 704 op/s; 19499/287283 objects misplaced (6.787%); 19 MiB/s, 296 objects/s recovering 2024-03-26T21:36:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/212282861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:20 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:36:21.109 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:36:21.109 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:36:21.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:21.150 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:21.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:21.290 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:21.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:21.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:21.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:21.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:21.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:21.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:36:21.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:21.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:36:21.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[29406]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T21:36:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:36:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:21 smithi055 ceph-mon[40587]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T21:36:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.844 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:21.920 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:21 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:21.920 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:21 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:36:21.920 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:21 smithi067 ceph-mon[33346]: osdmap e193: 8 total, 8 up, 7 in 2024-03-26T21:36:21.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:21.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 73 16 100128 100112 85.6973 61.5 0.0269396 0.011663 2024-03-26T21:36:21.989 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:21.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:22.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:22.125 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:22.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:22.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:22.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:22.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:22.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:22.550 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:22.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:22.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:22.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:22.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:22.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:22.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:22.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 74 16 101094 101078 85.3551 60.375 0.011077 0.0117103 2024-03-26T21:36:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:22 smithi067 ceph-mon[33346]: pgmap v327: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 19499/287283 objects misplaced (6.787%); 21 MiB/s, 339 objects/s recovering 2024-03-26T21:36:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:22 smithi055 ceph-mon[29406]: pgmap v327: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 19499/287283 objects misplaced (6.787%); 21 MiB/s, 339 objects/s recovering 2024-03-26T21:36:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:22 smithi055 ceph-mon[40587]: pgmap v327: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 19499/287283 objects misplaced (6.787%); 21 MiB/s, 339 objects/s recovering 2024-03-26T21:36:23.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:23.134 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:23.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:23.290 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:23.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:23.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:23.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:23.826 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:23.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:23.929 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:23 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:36:23.929 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:23 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:36:23.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:23 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:36:23.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:23 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:36:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:23.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:23.976 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:23 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:36:23.977 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:23 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:36:23.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 75 16 102027 102011 84.9945 58.3125 0.0226792 0.0117584 2024-03-26T21:36:24.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:24.109 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:24.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:24.242 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:24.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:24.365 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:36:24.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:24.426 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:36:24.446 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:36:24.446 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:36:24.475 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:36:24.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:24.520 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:36:24.534 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:24.560 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:36:24.602 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:36:24.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:24.651 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:36:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:24.669 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:36:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:24.694 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:36:24.702 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:36:24.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:24.787 INFO:teuthology.orchestra.run.smithi055.stderr:clear 5.1a pg_upmap mapping 2024-03-26T21:36:24.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:24.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:24.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:24.994 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 76 16 102844 102828 84.5414 51.0625 0.00645114 0.0118227 2024-03-26T21:36:25.031 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[29406]: pgmap v328: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 837 op/s; 17771/300360 objects misplaced (5.917%); 23 MiB/s, 369 objects/s recovering 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1463967192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3842030283' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[40587]: pgmap v328: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 837 op/s; 17771/300360 objects misplaced (5.917%); 23 MiB/s, 369 objects/s recovering 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1463967192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3842030283' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:24 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.133 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:25.170 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:24 smithi067 ceph-mon[33346]: pgmap v328: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 837 op/s; 17771/300360 objects misplaced (5.917%); 23 MiB/s, 369 objects/s recovering 2024-03-26T21:36:25.170 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:24 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1463967192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:25.170 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:24 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3842030283' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.170 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:24 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]: dispatch 2024-03-26T21:36:25.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:25.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:36:25.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.321 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:25.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:25.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:25.461 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:25.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:25.594 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:25.605 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:25.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:25.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:25.876 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:25.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:25.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:25 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]': finished 2024-03-26T21:36:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:25 smithi067 ceph-mon[33346]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T21:36:25.994 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 77 16 103793 103777 84.213 59.3125 0.0104681 0.0118693 2024-03-26T21:36:26.024 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:26.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:25 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]': finished 2024-03-26T21:36:26.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:25 smithi055 ceph-mon[40587]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T21:36:26.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:25 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1a"}]': finished 2024-03-26T21:36:26.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:25 smithi055 ceph-mon[29406]: osdmap e194: 8 total, 8 up, 7 in 2024-03-26T21:36:26.126 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:26.163 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:26.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:26.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:26.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:26.486 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:26.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:26.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:26.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:26.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:26.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:26.852 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:26.943 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[40587]: pgmap v330: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 17771/300360 objects misplaced (5.917%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:36:26.943 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:36:26.943 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[40587]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T21:36:26.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[29406]: pgmap v330: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 17771/300360 objects misplaced (5.917%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:36:26.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:36:26.944 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:26 smithi055 ceph-mon[29406]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T21:36:26.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:26.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:26 smithi067 ceph-mon[33346]: pgmap v330: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 545 op/s; 17771/300360 objects misplaced (5.917%); 15 MiB/s, 244 objects/s recovering 2024-03-26T21:36:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:26 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:36:26.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:26 smithi067 ceph-mon[33346]: osdmap e195: 8 total, 8 up, 7 in 2024-03-26T21:36:26.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:26.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 78 16 104719 104703 83.8754 57.875 0.0130379 0.0119168 2024-03-26T21:36:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:26.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:27.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:27.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:27.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:27.292 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:27.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:27.431 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:27.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:27.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:27.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:27.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:27.810 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:36:27.810 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:36:27.811 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:36:27.811 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:36:27.811 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:36:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:27.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:27.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:27 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:36:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:27.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 79 16 105703 105687 83.5921 61.5 0.0306668 0.0119561 2024-03-26T21:36:28.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:27 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:36:28.045 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:28.045 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:27 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:36:28.045 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:28.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:28.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:28.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:28.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:28.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:36:28.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:28.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:36:28.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:28.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:28.794 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:28.896 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:28.922 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:28.949 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:28.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[29406]: pgmap v332: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 16290/314307 objects misplaced (5.183%); 34 MiB/s, 544 objects/s recovering 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[40587]: pgmap v332: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 16290/314307 objects misplaced (5.183%); 34 MiB/s, 544 objects/s recovering 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:36:28.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:36:28.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:36:29.005617+0000 min lat: 0.0012867 max lat: 2.26773 avg lat: 0.0120118 2024-03-26T21:36:28.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:36:28.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 80 16 106550 106534 83.2067 52.9375 0.0153939 0.0120118 2024-03-26T21:36:29.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:29.109 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:28 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:36:29.109 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:28 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:36:29.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:28 smithi067 ceph-mon[33346]: pgmap v332: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 16290/314307 objects misplaced (5.183%); 34 MiB/s, 544 objects/s recovering 2024-03-26T21:36:29.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:36:29.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:36:29.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:29.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:29.263 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:29.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:29.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:29.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:29.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:29.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:29.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:29.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:29.877 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:36:29.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:29.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:29.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 81 16 107458 107442 82.88 56.75 0.00943141 0.0120602 2024-03-26T21:36:30.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:30.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:30.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[40587]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T21:36:30.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/236805700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[29406]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T21:36:30.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/236805700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:29 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:29 smithi067 ceph-mon[33346]: osdmap e196: 8 total, 8 up, 7 in 2024-03-26T21:36:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/236805700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:29 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:36:30.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:30.334 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:30.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:30.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:30.604 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:30.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.755 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:30.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:30.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:36:30.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:36:30.788 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-26T21:36:30.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd in 4 2024-03-26T21:36:30.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:30.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:30.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:30.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 82 16 108302 108286 82.5126 52.75 0.00976798 0.0121137 2024-03-26T21:36:31.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:31.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.177 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:31.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:30 smithi067 ceph-mon[33346]: pgmap v334: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:30 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:36:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:30 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:36:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:30 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:30 smithi067 ceph-mon[33346]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T21:36:31.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:31.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[29406]: pgmap v334: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:36:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:36:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[29406]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T21:36:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[40587]: pgmap v334: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:36:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:36:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:36:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:30 smithi055 ceph-mon[40587]: osdmap e197: 8 total, 8 up, 7 in 2024-03-26T21:36:31.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:31.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:31.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:31.580 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:31.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:31.736 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:31.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:31.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:31.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:31.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 83 16 109055 109039 82.0855 47.0625 0.00488286 0.0121746 2024-03-26T21:36:32.016 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:32.147 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:32.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:32.222 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:36:32.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:32.264 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:36:32.305 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:36:32.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:32.345 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:36:32.362 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:32.385 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:36:32.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:32.489 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:36:32.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:32.534 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:36:32.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: pgmap v336: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2110564985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: pgmap v336: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2110564985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:32.575 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:36:32.587 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:36:32.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:32.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: pgmap v336: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 48 MiB/s wr, 775 op/s; 16290/314307 objects misplaced (5.183%); 18 MiB/s, 280 objects/s recovering 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2110564985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:32.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:32.786 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:32.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:32.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:32.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:32.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 84 15 109719 109704 81.6031 41.5625 0.0122222 0.012249 2024-03-26T21:36:33.035 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:33.078 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:33.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:33.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:33.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:33.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:33.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:36:33.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:33.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:33.740 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:33.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:33 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:36:33.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:33 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:36:33.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:33 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:36:33.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:33 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:36:33.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:33.890 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:33.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:33 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:36:33.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:33 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:36:33.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:33.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 85 16 110438 110422 81.171 44.875 0.0291785 0.0123141 2024-03-26T21:36:34.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:34.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:34.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:34.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:34.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.439 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:34.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:34.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:34.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:34.731 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:34 smithi055 ceph-mon[29406]: pgmap v337: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 14424/327222 objects misplaced (4.408%); 30 MiB/s, 484 objects/s recovering 2024-03-26T21:36:34.731 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:34 smithi055 ceph-mon[40587]: pgmap v337: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 14424/327222 objects misplaced (4.408%); 30 MiB/s, 484 objects/s recovering 2024-03-26T21:36:34.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:34.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:34.962 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:34 smithi067 ceph-mon[33346]: pgmap v337: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 14424/327222 objects misplaced (4.408%); 30 MiB/s, 484 objects/s recovering 2024-03-26T21:36:34.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:34.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:34.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 86 16 111390 111374 80.919 59.5 0.00587466 0.0123526 2024-03-26T21:36:35.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:35.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:35.194 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:35.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:35.300 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:35.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:35.438 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:35.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:35.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:35.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:35.715 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:36:35.715 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:36:35.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:35.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:35.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:35.988 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:35.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:35.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 87 16 112262 112246 80.6153 54.5 0.0195629 0.0123987 2024-03-26T21:36:36.090 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:36.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:36.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:36.264 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:36.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:36.411 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:36.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:36.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:36.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:36.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:36:36.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: pgmap v338: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 14424/327222 objects misplaced (4.408%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:36:36.811 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: pgmap v338: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 14424/327222 objects misplaced (4.408%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:36:36.812 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:36 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:36:36.840 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:36.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:36:36.844 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:36:36.844 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: pgmap v338: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 14424/327222 objects misplaced (4.408%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:36:36.844 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: osdmap e198: 8 total, 8 up, 7 in 2024-03-26T21:36:36.844 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:36:36.844 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:36:36.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:36.993 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:36.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:36.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 88 16 113049 113033 80.2582 49.1875 0.0178763 0.0124531 2024-03-26T21:36:37.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:37.148 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:37.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:37.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:37.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:37.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:37.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:37.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:37.543 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:37.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:37.779 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:37.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:37.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:37 smithi055 ceph-mon[29406]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T21:36:37.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:37 smithi055 ceph-mon[40587]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T21:36:37.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:37.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:37.949 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:37.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:37 smithi067 ceph-mon[33346]: osdmap e199: 8 total, 8 up, 7 in 2024-03-26T21:36:37.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:37.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 89 16 113626 113610 79.7617 36.0625 0.0157089 0.0125037 2024-03-26T21:36:38.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:38.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:38.207 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:38.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:38.332 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:38.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:38.462 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:38.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:38.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.724 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:38.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:38 smithi055 ceph-mon[29406]: pgmap v341: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 12633/339267 objects misplaced (3.724%); 32 MiB/s, 509 objects/s recovering 2024-03-26T21:36:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:38 smithi055 ceph-mon[29406]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T21:36:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:38 smithi055 ceph-mon[40587]: pgmap v341: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 12633/339267 objects misplaced (3.724%); 32 MiB/s, 509 objects/s recovering 2024-03-26T21:36:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:38 smithi055 ceph-mon[40587]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T21:36:38.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:38.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:38.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:38 smithi067 ceph-mon[33346]: pgmap v341: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 12633/339267 objects misplaced (3.724%); 32 MiB/s, 509 objects/s recovering 2024-03-26T21:36:38.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:38 smithi067 ceph-mon[33346]: osdmap e200: 8 total, 8 up, 7 in 2024-03-26T21:36:38.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:38.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:38.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 90 6 113966 113960 79.1185 21.875 0.0288031 0.0126317 2024-03-26T21:36:39.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:39.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:36:39.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:39.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:39.350 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.376 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:39.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:39.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:39.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:39.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:39.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:39.828 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:39 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:36:39.829 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:39 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:36:39.829 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:39 smithi067 ceph-mon[33346]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T21:36:39.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:39.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:39.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:39.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total time run: 90.049 2024-03-26T21:36:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total writes made: 113966 2024-03-26T21:36:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Write size: 65536 2024-03-26T21:36:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object size: 65536 2024-03-26T21:36:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Bandwidth (MB/sec): 79.1 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Bandwidth: 39.5741 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max bandwidth (MB/sec): 195.562 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min bandwidth (MB/sec): 0 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average IOPS: 1265 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev IOPS: 633.197 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max IOPS: 3129 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min IOPS: 0 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average Latency(s): 0.012634 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Latency(s): 0.0302047 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max latency(s): 2.26773 2024-03-26T21:36:39.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min latency(s): 0.0012867 2024-03-26T21:36:40.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T21:36:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:40.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:40.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:36:40.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:36:40.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[40587]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T21:36:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:36:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:36:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:39 smithi055 ceph-mon[29406]: osdmap e201: 8 total, 8 up, 7 in 2024-03-26T21:36:40.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:40.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:40.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:40.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:40.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:40.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:40.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:40.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:40.715 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:40.817 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:40.854 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:40.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:40.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:40 smithi067 ceph-mon[33346]: 8.7 deep-scrub starts 2024-03-26T21:36:40.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:40 smithi067 ceph-mon[33346]: pgmap v344: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 12633/339267 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-26T21:36:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:40.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:41.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:40 smithi055 ceph-mon[29406]: 8.7 deep-scrub starts 2024-03-26T21:36:41.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:40 smithi055 ceph-mon[29406]: pgmap v344: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 12633/339267 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-26T21:36:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:40 smithi055 ceph-mon[40587]: 8.7 deep-scrub starts 2024-03-26T21:36:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:40 smithi055 ceph-mon[40587]: pgmap v344: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 12633/339267 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-26T21:36:41.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.133 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:41.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:41.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:41.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:41.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:41.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:41.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:41.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:41.673 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:41.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:41.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:41.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:42.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:42.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:42.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:42.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:42.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:42.440 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.478 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:42.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:42.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:42.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:42.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:42.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:42 smithi055 ceph-mon[40587]: pgmap v345: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 865 B/s rd, 42 MiB/s wr, 679 op/s; 12633/339267 objects misplaced (3.724%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:36:42.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:42 smithi055 ceph-mon[29406]: pgmap v345: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 865 B/s rd, 42 MiB/s wr, 679 op/s; 12633/339267 objects misplaced (3.724%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:36:42.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:42.872 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:42.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:42.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:42 smithi067 ceph-mon[33346]: pgmap v345: 148 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 143 active+clean; 6.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 865 B/s rd, 42 MiB/s wr, 679 op/s; 12633/339267 objects misplaced (3.724%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:36:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:43.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:43.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:43.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:43.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:43.374 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:43.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:43.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:43.530 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:43.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:43.657 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:43.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:43.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:43.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:43.930 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:43.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:44.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:36:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:36:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[40587]: 4.1d scrub starts 2024-03-26T21:36:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[40587]: 4.1d scrub ok 2024-03-26T21:36:44.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[40587]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T21:36:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:36:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:36:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[29406]: 4.1d scrub starts 2024-03-26T21:36:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[29406]: 4.1d scrub ok 2024-03-26T21:36:44.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:43 smithi055 ceph-mon[29406]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T21:36:44.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:44.186 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:43 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:36:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:43 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:36:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:43 smithi067 ceph-mon[33346]: 4.1d scrub starts 2024-03-26T21:36:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:43 smithi067 ceph-mon[33346]: 4.1d scrub ok 2024-03-26T21:36:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:43 smithi067 ceph-mon[33346]: osdmap e202: 8 total, 8 up, 7 in 2024-03-26T21:36:44.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:44.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:44.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:44.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:44.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:44.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:44.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.839 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:44.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:44.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:44 smithi055 ceph-mon[40587]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 419 op/s; 10843/338190 objects misplaced (3.206%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:36:45.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:44 smithi055 ceph-mon[40587]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T21:36:45.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:44 smithi055 ceph-mon[29406]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 419 op/s; 10843/338190 objects misplaced (3.206%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:36:45.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:44 smithi055 ceph-mon[29406]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T21:36:45.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:45.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:45.234 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:44 smithi067 ceph-mon[33346]: pgmap v347: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 11 MiB/s wr, 419 op/s; 10843/338190 objects misplaced (3.206%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:36:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:44 smithi067 ceph-mon[33346]: osdmap e203: 8 total, 8 up, 7 in 2024-03-26T21:36:45.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:45.469 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:45.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:45.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:45.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:45.754 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:45.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:45.879 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:45.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:45.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:46.021 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:46.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:45 smithi055 ceph-mon[40587]: 8.1 scrub starts 2024-03-26T21:36:46.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:45 smithi055 ceph-mon[29406]: 8.1 scrub starts 2024-03-26T21:36:46.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:46.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:45 smithi067 ceph-mon[33346]: 8.1 scrub starts 2024-03-26T21:36:46.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.283 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:46.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:46.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:46.420 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:46.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:46.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:46.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:46.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:46.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:46.836 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:46.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:46.968 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:47.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:46 smithi055 ceph-mon[40587]: pgmap v349: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 907 B/s rd, 9.9 MiB/s wr, 371 op/s; 10843/338190 objects misplaced (3.206%); 17 MiB/s, 264 objects/s recovering 2024-03-26T21:36:47.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:47.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:46 smithi055 ceph-mon[29406]: pgmap v349: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 907 B/s rd, 9.9 MiB/s wr, 371 op/s; 10843/338190 objects misplaced (3.206%); 17 MiB/s, 264 objects/s recovering 2024-03-26T21:36:47.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:47.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:47.104 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:47.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:46 smithi067 ceph-mon[33346]: pgmap v349: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 1 active+clean+remapped, 143 active+clean; 6.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 907 B/s rd, 9.9 MiB/s wr, 371 op/s; 10843/338190 objects misplaced (3.206%); 17 MiB/s, 264 objects/s recovering 2024-03-26T21:36:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:36:47.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:47.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.365 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:47.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:47.466 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:47.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:47.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:47.639 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:47.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:47.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:47.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:47.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:48.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:48.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:36:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:36:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[29406]: 8.1 scrub ok 2024-03-26T21:36:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:48.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:36:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:36:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[40587]: 8.1 scrub ok 2024-03-26T21:36:48.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:48.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:47 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:36:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:47 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:36:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:47 smithi067 ceph-mon[33346]: 8.1 scrub ok 2024-03-26T21:36:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:48.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:48.284 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:48.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.427 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:48.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:48.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:48.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:48.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:48.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:48.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:48.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:48.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:49.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:49.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:48 smithi055 ceph-mon[40587]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:49.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:48 smithi055 ceph-mon[29406]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:49.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:49.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:48 smithi067 ceph-mon[33346]: pgmap v350: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:49.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:49.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:49.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:49.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:49.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:49.839 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:49.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:49.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:50.004 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:49 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:50.004 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:49 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:50.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:50.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:50.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:49 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:36:50.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:50.294 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:50.295 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.348 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:50.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:50.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:50.478 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:50.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:36:50.289+0000 7fb1ff446700 -1 mon.a@0(leader).osd e203 definitely_dead 0 2024-03-26T21:36:50.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:50.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:50.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:50.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:50.845 INFO:teuthology.orchestra.run.smithi055.stderr:marked in osd.4. 2024-03-26T21:36:50.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3657195895' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:50.881 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3657195895' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:50.882 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:50 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:50.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:50.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:51.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:51.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:51.156 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: pgmap v351: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 8.4 MiB/s wr, 674 op/s; 10084/329568 objects misplaced (3.060%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3657195895' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:36:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:50 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:36:51.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:51.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:51.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:51.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:51.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:51.664 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:51.681 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:36:51.765 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:51.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:51.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:51.919 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:51.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:51.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T21:36:51.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4222412327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:51.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:36:51.927 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[40587]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4222412327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:51.928 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:36:51.929 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:51 smithi055 ceph-mon[29406]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:36:52.020 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:52.047 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:52.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:52.174 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4222412327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:36:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:36:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:36:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:36:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:36:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:36:52.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:51 smithi067 ceph-mon[33346]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T21:36:52.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:52.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:52.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.436 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:52.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:52.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:52.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:52.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:52.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:52.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:52.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:52.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:52.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:52 smithi055 ceph-mon[29406]: pgmap v353: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 359 op/s; 10084/329568 objects misplaced (3.060%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T21:36:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:52 smithi055 ceph-mon[29406]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:36:53.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:52 smithi055 ceph-mon[40587]: pgmap v353: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 359 op/s; 10084/329568 objects misplaced (3.060%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T21:36:53.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:52 smithi055 ceph-mon[40587]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:36:53.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:53.104 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:53.206 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:52 smithi067 ceph-mon[33346]: pgmap v353: 148 pgs: 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 144 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 359 op/s; 10084/329568 objects misplaced (3.060%); 5.9 MiB/s, 94 objects/s recovering 2024-03-26T21:36:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:52 smithi067 ceph-mon[33346]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T21:36:53.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:53.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:53.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:53.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:53.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:53.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:53.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:53.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:53.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:53.982 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-26T21:36:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:53 smithi055 ceph-mon[40587]: 8.2 starting backfill to osd.4 from (179'2838,201'7305] MIN to 203'7605 2024-03-26T21:36:53.982 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:53 smithi055 ceph-mon[40587]: 8.b starting backfill to osd.4 from (179'2722,199'7155] MIN to 203'7616 2024-03-26T21:36:53.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:53 smithi055 ceph-mon[29406]: 8.2 starting backfill to osd.4 from (179'2838,201'7305] MIN to 203'7605 2024-03-26T21:36:53.982 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:53 smithi055 ceph-mon[29406]: 8.b starting backfill to osd.4 from (179'2722,199'7155] MIN to 203'7616 2024-03-26T21:36:54.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:54.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:54.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:53 smithi067 ceph-mon[33346]: 8.2 starting backfill to osd.4 from (179'2838,201'7305] MIN to 203'7605 2024-03-26T21:36:54.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:53 smithi067 ceph-mon[33346]: 8.b starting backfill to osd.4 from (179'2722,199'7155] MIN to 203'7616 2024-03-26T21:36:54.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:54.391 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:54.493 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.531 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:54.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:54.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:54.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:54.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:36:54.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:54.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:54.930 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:54.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:55.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:54 smithi055 ceph-mon[40587]: pgmap v356: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 110 objects/s recovering 2024-03-26T21:36:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:54 smithi055 ceph-mon[29406]: pgmap v356: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 110 objects/s recovering 2024-03-26T21:36:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:55.170 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:55.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:54 smithi067 ceph-mon[33346]: pgmap v356: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 110 objects/s recovering 2024-03-26T21:36:55.304 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:55.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:55.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:55.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:55.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:55.731 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:55.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:55.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:55.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:56.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:36:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:36:56.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:56.259 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:56.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:56.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:56.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:56.532 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:56.633 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:56.669 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:56.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:56.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:56.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:56.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:56.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:57.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:36:57.082 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:36:57.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:36:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:36:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:56 smithi067 ceph-mon[33346]: pgmap v357: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 111 objects/s recovering 2024-03-26T21:36:57.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:36:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:36:57.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:56 smithi055 ceph-mon[40587]: pgmap v357: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 111 objects/s recovering 2024-03-26T21:36:57.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:56 smithi055 ceph-mon[29406]: pgmap v357: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 4 remapped+peering, 3 activating, 18 peering, 121 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 452 op/s; 2067/321444 objects misplaced (0.643%); 6.7 MiB/s, 111 objects/s recovering 2024-03-26T21:36:57.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:57.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:57.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:57.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:57.600 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:57.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:57.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:57.790 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:57.810 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:36:57.810 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:36:57.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:57.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:58.001 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:58.002 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:58.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:58.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:58.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:58.200 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:36:58.302 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:58.340 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:58.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:58.480 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:36:58.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:58.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:58.590 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:36:58.622 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:58.626 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:36:58.667 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:36:58.710 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:36:58.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:58.757 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:36:58.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:58.798 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:36:58.840 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:36:58.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:58.874 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:36:58.877 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:36:58.877 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:36:58.877 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:36:58.878 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:36:58.878 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:36:58.878 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:36:58.878 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:36:58.878 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:36:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:58.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:58.982 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:36:58.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:36:58.982 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-26T21:36:58.982 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:36:58.982 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:36:58.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-26T21:36:59.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:59.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:36:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.188 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:59.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:36:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:58 smithi067 ceph-mon[33346]: pgmap v358: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 905 op/s; 18551/311649 objects misplaced (5.953%); 14 MiB/s, 234 objects/s recovering 2024-03-26T21:36:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:58 smithi067 ceph-mon[33346]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:36:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:36:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4187608420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:59.250 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:36:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:36:59.250 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[40587]: pgmap v358: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 905 op/s; 18551/311649 objects misplaced (5.953%); 14 MiB/s, 234 objects/s recovering 2024-03-26T21:36:59.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[40587]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:36:59.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4187608420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:59.258 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[29406]: pgmap v358: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 905 op/s; 18551/311649 objects misplaced (5.953%); 14 MiB/s, 234 objects/s recovering 2024-03-26T21:36:59.258 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[29406]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:36:59.258 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:36:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4187608420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:36:59.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:59.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:36:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.340 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:59.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:59.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:36:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:36:59.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:36:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:59.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:36:59.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:36:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.758 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:36:59.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:36:59.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:36:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:36:59.898 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:36:59.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:36:59.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:00.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:00.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:00.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:00.308 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:00.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:00.446 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:00.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:00.581 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:00.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:00.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:00.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:00.876 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:00.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:00.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:00.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.021 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:01.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:01.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:01.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:00 smithi067 ceph-mon[33346]: pgmap v359: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 18551/311649 objects misplaced (5.953%); 12 MiB/s, 193 objects/s recovering 2024-03-26T21:37:01.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:01.297 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:01.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:00 smithi055 ceph-mon[29406]: pgmap v359: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 18551/311649 objects misplaced (5.953%); 12 MiB/s, 193 objects/s recovering 2024-03-26T21:37:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:00 smithi055 ceph-mon[40587]: pgmap v359: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 18551/311649 objects misplaced (5.953%); 12 MiB/s, 193 objects/s recovering 2024-03-26T21:37:01.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:01.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:01.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:01.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:01.833 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:01.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:01.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:37:01.934 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:02.001 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:02.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:02.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:02.110 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:02.110 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:02.110 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[40587]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[29406]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:02.111 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:02.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:01 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:01 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:01 smithi067 ceph-mon[33346]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T21:37:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:02.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:02.286 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:02.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:02.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:02.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:02.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:02.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:02.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:02.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:02.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:02.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:02.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:03.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:03.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:03.156 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[40587]: pgmap v361: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 372 op/s; 18551/311649 objects misplaced (5.953%); 6.2 MiB/s, 100 objects/s recovering 2024-03-26T21:37:03.157 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:03.157 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:03.157 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[29406]: pgmap v361: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 372 op/s; 18551/311649 objects misplaced (5.953%); 6.2 MiB/s, 100 objects/s recovering 2024-03-26T21:37:03.157 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:03.157 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:02 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:03.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:02 smithi067 ceph-mon[33346]: pgmap v361: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 372 op/s; 18551/311649 objects misplaced (5.953%); 6.2 MiB/s, 100 objects/s recovering 2024-03-26T21:37:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:02 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:02 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:03.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:03.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:03.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:03.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:03.530 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:03.633 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:03.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:03.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:03.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:03.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:03.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:04.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:04.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:04.213 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:03 smithi067 ceph-mon[33346]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:37:04.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:03 smithi055 ceph-mon[29406]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:37:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:03 smithi055 ceph-mon[40587]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T21:37:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.350 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:04.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:04.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:04.536 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:04.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:04.671 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:04.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:04.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:04.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:04.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:05.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:05.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:04 smithi067 ceph-mon[33346]: pgmap v363: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 17167/298806 objects misplaced (5.745%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:37:05.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:04 smithi055 ceph-mon[40587]: pgmap v363: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 17167/298806 objects misplaced (5.745%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:37:05.251 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:04 smithi055 ceph-mon[29406]: pgmap v363: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 944 op/s; 17167/298806 objects misplaced (5.745%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:37:05.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.349 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:05.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:05.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:05.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:05.790 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:05.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:05.924 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:06.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:06.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:06.003 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[29406]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:37:06.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:06.036 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:37:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.089 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:06.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:06.190 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.226 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:06.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[40587]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:37:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:06.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:06.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:06.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:06 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:06 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:06 smithi067 ceph-mon[33346]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T21:37:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:06.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:06.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:06.641 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:37:06.641 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T21:37:06.641 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:37:06.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:06.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:06.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:06.906 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:06.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:07.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:07.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:07.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[40587]: pgmap v365: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 714 op/s; 17167/298806 objects misplaced (5.745%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[40587]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[29406]: pgmap v365: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 714 op/s; 17167/298806 objects misplaced (5.745%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:07.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:07 smithi055 ceph-mon[29406]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:37:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:07 smithi067 ceph-mon[33346]: pgmap v365: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 714 op/s; 17167/298806 objects misplaced (5.745%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T21:37:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:07 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/897004013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:07 smithi067 ceph-mon[33346]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T21:37:07.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:07.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.325 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:07.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:07.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:07.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:07.597 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:07.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:07.742 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:07.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:07.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:07.889 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:37:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:37:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:37:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:37:07.889 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:37:07.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:08.027 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:08.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:08.167 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:08.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:08.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:08.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:08.379 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:08.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:08.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:08.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:08.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:08.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:08.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:08.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:08.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:09.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:09.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:09 smithi055 ceph-mon[40587]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 15602/285768 objects misplaced (5.460%); 19 MiB/s, 306 objects/s recovering 2024-03-26T21:37:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:09 smithi055 ceph-mon[29406]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 15602/285768 objects misplaced (5.460%); 19 MiB/s, 306 objects/s recovering 2024-03-26T21:37:09.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:37:09.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:09.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:37:09.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:09.317 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:09.390 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:09 smithi067 ceph-mon[33346]: pgmap v367: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s; 15602/285768 objects misplaced (5.460%); 19 MiB/s, 306 objects/s recovering 2024-03-26T21:37:09.419 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:09.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:09.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:09.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:09.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:09.766 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:09.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:09.903 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:09.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:10.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:10.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:10.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:10.181 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:10.283 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:10.324 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:10.411 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:10.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:10.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:10.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:10.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:10.632 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:10.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:10.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:10.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:10.907 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:10.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:11.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:11.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:11.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:11.187 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:11.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:11 smithi055 ceph-mon[40587]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 673 op/s; 15602/285768 objects misplaced (5.460%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T21:37:11.226 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:11 smithi055 ceph-mon[29406]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 673 op/s; 15602/285768 objects misplaced (5.460%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T21:37:11.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:11.322 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:11.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.462 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:11 smithi067 ceph-mon[33346]: pgmap v368: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 792 B/s rd, 0 B/s wr, 673 op/s; 15602/285768 objects misplaced (5.460%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T21:37:11.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:11.574 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:37:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:11.626 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:37:11.672 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:37:11.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:11.711 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:37:11.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:11.757 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:37:11.798 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:37:11.835 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:37:11.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:11.876 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:11.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:11.881 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:37:11.883 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:37:11.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:12.015 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:12.117 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:12.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:12.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:12.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:12.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2625327692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2625327692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:12.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:12.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2625327692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:12.533 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:12.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:12.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:12.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.840 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:12.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:12.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:12.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:13.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.114 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:13.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:13.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:13.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:13 smithi055 ceph-mon[29406]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 543 op/s; 15602/285768 objects misplaced (5.460%); 7.8 MiB/s, 124 objects/s recovering 2024-03-26T21:37:13.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:13 smithi055 ceph-mon[40587]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 543 op/s; 15602/285768 objects misplaced (5.460%); 7.8 MiB/s, 124 objects/s recovering 2024-03-26T21:37:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:13.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:13.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:13.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:37:13.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:37:13.413 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-26T21:37:13.413 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:37:13.413 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:37:13.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-26T21:37:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:13 smithi067 ceph-mon[33346]: pgmap v369: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 543 op/s; 15602/285768 objects misplaced (5.460%); 7.8 MiB/s, 124 objects/s recovering 2024-03-26T21:37:13.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.543 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:13.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:13.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:13.779 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:13.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:13.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:13.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:13.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:14.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:14.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:14.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:14.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:14.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:14.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.556 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:14.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:14.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:14.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:14.714 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:37:14.714 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:37:14.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:14.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:15.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:15.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:15.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:15.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:15 smithi055 ceph-mon[29406]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 13991/270018 objects misplaced (5.182%); 14 MiB/s, 226 objects/s recovering 2024-03-26T21:37:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:15 smithi055 ceph-mon[40587]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 13991/270018 objects misplaced (5.182%); 14 MiB/s, 226 objects/s recovering 2024-03-26T21:37:15.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:15.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:15 smithi067 ceph-mon[33346]: pgmap v370: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 13991/270018 objects misplaced (5.182%); 14 MiB/s, 226 objects/s recovering 2024-03-26T21:37:15.499 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:15.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:15.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:37:15.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:15.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:15.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:16.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:16.052 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:16.103 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:37:16.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.188 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:16.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:16.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:16.326 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:16.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:16.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:16 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:37:16.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:16 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:37:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:16 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:37:16.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:16.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:16.710 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:16.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:16.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:16.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:16.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:16.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:17.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:17.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:17.168 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:17.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s; 13991/270018 objects misplaced (5.182%); 12 MiB/s, 190 objects/s recovering 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: 8.7 scrub starts 2024-03-26T21:37:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:17.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:17.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:17.330 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s; 13991/270018 objects misplaced (5.182%); 12 MiB/s, 190 objects/s recovering 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: 8.7 scrub starts 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: pgmap v371: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s; 13991/270018 objects misplaced (5.182%); 12 MiB/s, 190 objects/s recovering 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1565694176' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: 8.7 scrub starts 2024-03-26T21:37:17.331 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:17.343 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:17.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:17.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:17.588 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:17.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:17.732 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:17.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:17.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:17.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:17.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:17.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:18.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:18.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:18.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:37:18.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:18.281 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:18.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.420 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:18.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:18.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:18.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:18.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:18.680 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:18.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:18.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:18.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:19.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:19.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:19.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:19.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:19.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[29406]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[29406]: 8.7 scrub ok 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[29406]: 8.3 scrub starts 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[40587]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[40587]: 8.7 scrub ok 2024-03-26T21:37:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:19 smithi055 ceph-mon[40587]: 8.3 scrub starts 2024-03-26T21:37:19.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:19.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:19.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:37:19.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:19 smithi067 ceph-mon[33346]: pgmap v373: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:19 smithi067 ceph-mon[33346]: 8.7 scrub ok 2024-03-26T21:37:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:19 smithi067 ceph-mon[33346]: 8.3 scrub starts 2024-03-26T21:37:19.504 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:19.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:19.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:19.785 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:19.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:19.928 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:19.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:20.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:20.073 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:20.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:20.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:20 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:37:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:20 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:37:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:20 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:37:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:20 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:37:20.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:20.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:20.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:20 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:37:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:20 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:37:20.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:20.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:20.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:20.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:20.770 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:20.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:20.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:20.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:21.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:21.072 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:21.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.214 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:21.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:21 smithi055 ceph-mon[29406]: pgmap v374: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:21 smithi055 ceph-mon[29406]: 8.3 scrub ok 2024-03-26T21:37:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:21 smithi055 ceph-mon[40587]: pgmap v374: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:21 smithi055 ceph-mon[40587]: 8.3 scrub ok 2024-03-26T21:37:21.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:21.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:21.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:21 smithi067 ceph-mon[33346]: pgmap v374: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:21 smithi067 ceph-mon[33346]: 8.3 scrub ok 2024-03-26T21:37:21.492 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:21.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:21.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:21.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:21.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:21.952 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:22.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:22.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:22.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:22.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:22.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:22.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:22.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:22.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:22.591 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:22.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:22.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:22.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:22.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:22.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:22.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:22.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:22 smithi067 ceph-mon[33346]: 8.a scrub starts 2024-03-26T21:37:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:22 smithi067 ceph-mon[33346]: pgmap v375: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:22 smithi067 ceph-mon[33346]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:37:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[40587]: 8.a scrub starts 2024-03-26T21:37:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[40587]: pgmap v375: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[40587]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:37:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[29406]: 8.a scrub starts 2024-03-26T21:37:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[29406]: pgmap v375: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 12289/254058 objects misplaced (4.837%); 12 MiB/s, 196 objects/s recovering 2024-03-26T21:37:23.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:22 smithi055 ceph-mon[29406]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T21:37:23.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:23.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:23.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:23.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:23.362 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:23.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:23.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:23.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:23.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:23.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:23.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:23.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:23.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:23 smithi067 ceph-mon[33346]: 8.a scrub ok 2024-03-26T21:37:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:23 smithi067 ceph-mon[33346]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:37:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:24.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:23 smithi055 ceph-mon[40587]: 8.a scrub ok 2024-03-26T21:37:24.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:23 smithi055 ceph-mon[40587]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:37:24.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:23 smithi055 ceph-mon[29406]: 8.a scrub ok 2024-03-26T21:37:24.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:23 smithi055 ceph-mon[29406]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T21:37:24.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:24.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:24.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:24.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.412 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:24.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:24.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:24.550 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:24.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:24.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:24.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:24.818 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:24.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:24.948 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:24.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: pgmap v378: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 10621/240294 objects misplaced (4.420%); 19 MiB/s, 412 objects/s recovering 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:37:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:37:24.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:37:24.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:24 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:37:25.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: pgmap v378: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 10621/240294 objects misplaced (4.420%); 19 MiB/s, 412 objects/s recovering 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:37:25.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: pgmap v378: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 10621/240294 objects misplaced (4.420%); 19 MiB/s, 412 objects/s recovering 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:37:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:24 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:37:25.073 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:25.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:25.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.337 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:25.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:25.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:25.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:25.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:25.596 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:25.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:25.722 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:25.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:25.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:25 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:37:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:25 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:37:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:25.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:25.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:26.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:25 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:37:26.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:25 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:37:26.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:25 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:37:26.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:25 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:37:26.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:26.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:26.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:26.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:26.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:26.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:26.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:26.516 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:26.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:26.641 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:26.742 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:26.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:26.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:26.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:26.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:26 smithi067 ceph-mon[33346]: pgmap v380: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 764 op/s; 10621/240294 objects misplaced (4.420%); 12 MiB/s, 329 objects/s recovering 2024-03-26T21:37:26.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:27.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.040 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:26 smithi055 ceph-mon[40587]: pgmap v380: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 764 op/s; 10621/240294 objects misplaced (4.420%); 12 MiB/s, 329 objects/s recovering 2024-03-26T21:37:27.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:26 smithi055 ceph-mon[29406]: pgmap v380: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 764 op/s; 10621/240294 objects misplaced (4.420%); 12 MiB/s, 329 objects/s recovering 2024-03-26T21:37:27.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:27.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:27.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:27.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:27.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:27.447 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:27.549 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:27.590 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:27.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:27.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:27.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:27.874 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:27.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:27.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:28.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:28.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:28.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:28.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:28.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:28.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:28.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:28.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:28.379 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:28.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:28.503 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:28.605 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:28.629 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:28.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:28.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:28.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:28.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:28.987 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s; 4971/231228 objects misplaced (2.150%); 20 MiB/s, 459 objects/s recovering 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s; 4971/231228 objects misplaced (2.150%); 20 MiB/s, 459 objects/s recovering 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:37:28.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s; 4971/231228 objects misplaced (2.150%); 20 MiB/s, 459 objects/s recovering 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:37:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:28 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:37:28.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:29.036 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:29.137 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:29.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:29.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:29.308 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:29.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:29.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:29.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:29.579 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:29.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:29.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:29.853 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:29.873 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:37:29.873 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T21:37:29.873 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:37:29.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:29.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:30.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:37:30.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:37:30.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:29 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:37:30.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:30.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:30.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:37:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:37:30.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:37:30.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:29 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:37:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:30.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:30.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:30.395 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:30.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:30.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:30.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.801 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:30.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:30.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:30.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:30.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:31.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:31.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:37:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:37:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:37:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:37:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:30 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:37:31.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:31.192 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:37:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:37:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:37:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:37:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:37:31.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:30 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:37:31.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:31.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:31.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:31.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:31.568 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:31.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:31.695 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:31.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:31.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:31.963 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:31.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:32.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:37:32.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:37:32.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:31 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:32.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:37:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:31 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:32.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:32.264 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:32.264 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:32.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:32.488 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:32.533 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:32.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:32.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:32.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:32.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.6 scrub starts 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.6 scrub ok 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:37:32.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.6 scrub starts 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.6 scrub ok 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:37:32.934 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:37:32.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:37:32.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:37:32.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:37:32.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:32 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:37:32.951 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:32.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:33.053 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:33.092 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:33.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 145 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 504 op/s; 4971/231228 objects misplaced (2.150%); 7.5 MiB/s, 129 objects/s recovering 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.6 scrub starts 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.6 scrub ok 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:37:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:37:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:37:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:37:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:37:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:32 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:37:33.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:33.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:33.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:33.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:33.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:33.646 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:33.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:33.781 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:33.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:33.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:34.021 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:34.056 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:34.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:34.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:37:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:37:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:37:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:37:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:37:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:33 smithi067 ceph-mon[33346]: 8.9 scrub starts 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:37:34.288 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[40587]: 8.9 scrub starts 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:37:34.289 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:33 smithi055 ceph-mon[29406]: 8.9 scrub starts 2024-03-26T21:37:34.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.343 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:34.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:34.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:34.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:34.501 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:37:34.501 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:37:34.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:34.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:34.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.760 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:34.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:34.861 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:34.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:34.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:34.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s; 4596/219459 objects misplaced (2.094%); 1.6 MiB/s, 112 objects/s recovering 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 8.9 scrub ok 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:37:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s; 4596/219459 objects misplaced (2.094%); 1.6 MiB/s, 112 objects/s recovering 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 8.9 scrub ok 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:37:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:34 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:37:35.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:35.169 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: pgmap v389: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s; 4596/219459 objects misplaced (2.094%); 1.6 MiB/s, 112 objects/s recovering 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:37:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 8.9 scrub ok 2024-03-26T21:37:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:37:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:37:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:37:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:34 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:37:35.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:35.307 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:35.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:37:35.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.444 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:35.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:35.545 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:35.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:35.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:35.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:35.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:35.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:35.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:35.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:36.015 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:37:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:37:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:37:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:37:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:37:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:37:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:37:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:35 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:37:36.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:36.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:35 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:37:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:35 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:37:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:35 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:37:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:35 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:37:36.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:36.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:36.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:36.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:36.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:36.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:36.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:36.843 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:36.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:36.989 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:36.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:37.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:37.116 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: pgmap v390: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 654 op/s; 4596/219459 objects misplaced (2.094%); 1.3 MiB/s, 87 objects/s recovering 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: 8.5 scrub starts 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:37:37.197 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: pgmap v390: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 654 op/s; 4596/219459 objects misplaced (2.094%); 1.3 MiB/s, 87 objects/s recovering 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: 8.5 scrub starts 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:37:37.198 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:36 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:37:37.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:37.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: pgmap v390: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 654 op/s; 4596/219459 objects misplaced (2.094%); 1.3 MiB/s, 87 objects/s recovering 2024-03-26T21:37:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:37:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:37:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: 8.5 scrub starts 2024-03-26T21:37:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:37:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:36 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:37:37.252 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:37.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:37.395 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:37.497 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.542 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:37.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:37.643 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:37.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:37.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:37.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:37.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:37.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:38.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.101 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:38.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.18 scrub starts 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.18 scrub ok 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 8.5 scrub ok 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:37:38.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.18 scrub starts 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.18 scrub ok 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 8.5 scrub ok 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:37:38.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:37 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:37:38.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:38.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.18 scrub starts 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.18 scrub ok 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 8.5 scrub ok 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:37:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:37 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:37:38.299 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:37:38.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:38.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:38.368 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:37:38.407 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:37:38.446 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:37:38.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:38.491 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:37:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:38.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:38.538 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:37:38.578 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:37:38.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:38.619 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:37:38.625 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:37:38.625 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:37:38.626 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:37:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:38.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:38.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:38.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:38.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:38.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:38.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:38.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[29406]: pgmap v391: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 4211/202473 objects misplaced (2.080%); 1.2 MiB/s, 79 objects/s recovering 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3319015498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[40587]: pgmap v391: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 4211/202473 objects misplaced (2.080%); 1.2 MiB/s, 79 objects/s recovering 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:37:38.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3319015498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:39.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:39.080 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:38 smithi067 ceph-mon[33346]: pgmap v391: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 4211/202473 objects misplaced (2.080%); 1.2 MiB/s, 79 objects/s recovering 2024-03-26T21:37:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:38 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:37:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:38 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:37:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3319015498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:39.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:39.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:39.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:39.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:39.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.516 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:39.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:39.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:39.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:39.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:39.781 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:39.782 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:39.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:39.914 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:39.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:40.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:40.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:40.142 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:37:40.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:39 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:37:40.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:40.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:39 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:37:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:39 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:37:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:39 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:37:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:39 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:37:40.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:40.314 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:40.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:40.453 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:40.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:40.586 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:40.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:40.730 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:40.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:40.861 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:40.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:40.912 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: pgmap v392: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 4211/202473 objects misplaced (2.080%); 984 KiB/s, 65 objects/s recovering 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:37:40.931 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: pgmap v392: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 4211/202473 objects misplaced (2.080%); 984 KiB/s, 65 objects/s recovering 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:37:40.932 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:40 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:40.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.000 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:41.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:41.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.131 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:41.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:41.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:37:41.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:37:41.172 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-26T21:37:41.172 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:37:41.172 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:37:41.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: pgmap v392: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 4211/202473 objects misplaced (2.080%); 984 KiB/s, 65 objects/s recovering 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:37:41.204 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:40 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:37:41.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:41.276 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:41.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:41.421 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:41.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:37:41.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:37:41.555 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:41.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:41.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:41.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:41.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:41.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:37:41.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:41 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:37:42.019 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:42.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:37:42.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/368729617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:37:42.250 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:41 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:37:42.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:42.297 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:42.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:42.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:42.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:42.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:42.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:42.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:42.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:42.969 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:42.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:42.987 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: pgmap v394: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 656 op/s; 4211/202473 objects misplaced (2.080%) 2024-03-26T21:37:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:37:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:37:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:37:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:37:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:37:42.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:42 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:37:43.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:43.089 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:37:43.090 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:37:43.090 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:37:43.090 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:37:43.090 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:37:43.111 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: pgmap v394: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 656 op/s; 4211/202473 objects misplaced (2.080%) 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: pgmap v394: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 656 op/s; 4211/202473 objects misplaced (2.080%) 2024-03-26T21:37:43.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:37:43.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:37:43.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:37:43.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:37:43.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:37:43.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:42 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:37:43.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:43.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:43.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:43.548 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T21:37:43.649 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:37:43.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:43.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:43.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:43.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:43.952 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:43.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:43.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:37:44.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:37:44.054 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:44.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:44.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:37:44.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:37:44.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:37:44.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:37:44.216 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:37:44.216 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:37:44.216 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:37:44.216 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:43 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:37:44.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:44.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:43 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:37:44.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:43 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:37:44.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:43 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:37:44.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:43 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:37:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:44.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:44.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:44.542 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:44.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:44.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:44.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:44.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:44.985 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:44.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:45.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:45.125 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:45.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 4.12 deep-scrub starts 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 4.12 deep-scrub ok 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:37:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:37:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:44 smithi067 ceph-mon[33346]: 8.e scrub starts 2024-03-26T21:37:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 4.12 deep-scrub starts 2024-03-26T21:37:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 4.12 deep-scrub ok 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[40587]: 8.e scrub starts 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 4.12 deep-scrub starts 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 4.12 deep-scrub ok 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:37:45.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:44 smithi055 ceph-mon[29406]: 8.e scrub starts 2024-03-26T21:37:45.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.390 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:45.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:45.469 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:45.478 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:45.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:45.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:45.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:45.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:45.962 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:45.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:46.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:46.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:46.229 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:45 smithi067 ceph-mon[33346]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:37:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:45 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:37:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:45 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[29406]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[40587]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:37:46.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:45 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:37:46.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:46.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:46.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:46.639 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:46.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:46.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:46.892 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:37:46.893 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:37:46.909 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:37:46.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:46.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:37:46.919 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:37:46.920 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:37:47.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:47.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:47.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:47.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: pgmap v397: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: 8.e scrub ok 2024-03-26T21:37:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:37:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:37:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2995308381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:47.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:47.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: pgmap v397: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:47.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: 8.e scrub ok 2024-03-26T21:37:47.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2995308381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: pgmap v397: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 3882/185808 objects misplaced (2.089%) 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: 8.e scrub ok 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:37:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:37:47.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2995308381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:37:47.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:47.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:37:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.317 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:47.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:47.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:47.466 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:47.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:47.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:47.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:47.744 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:47.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:47.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:47.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:47.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:48.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:37:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:37:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:37:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:37:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:48.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:37:48.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:47 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:37:48.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:48.272 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:37:48.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:48.274 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:37:48.274 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:47 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:37:48.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:48.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:48.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:48.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:48.577 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:48.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:48.713 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:48.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:48.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:48.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:48.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:49.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:49.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 3882/173148 objects misplaced (2.242%) 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: 8.8 scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[29406]: 8.6 deep-scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 3882/173148 objects misplaced (2.242%) 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:49.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: 8.8 scrub starts 2024-03-26T21:37:49.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:48 smithi055 ceph-mon[40587]: 8.6 deep-scrub starts 2024-03-26T21:37:49.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:49.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:49.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 3882/173148 objects misplaced (2.242%) 2024-03-26T21:37:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:37:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:37:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: 8.8 scrub starts 2024-03-26T21:37:49.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:48 smithi067 ceph-mon[33346]: 8.6 deep-scrub starts 2024-03-26T21:37:49.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:49.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:49.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:49.395 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:37:49.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:37:49.395 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-03-26T21:37:49.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:37:49.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:49.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:49.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:49.634 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.676 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:49.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:49.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:49.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:49.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:49.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:50.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:50.074 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:50.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:37:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:49 smithi067 ceph-mon[33346]: 8.8 scrub ok 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:49 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[29406]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:37:50.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[29406]: 8.8 scrub ok 2024-03-26T21:37:50.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:49 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:37:50.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:37:50.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:37:50.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:37:50.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[40587]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T21:37:50.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:50 smithi055 ceph-mon[40587]: 8.8 scrub ok 2024-03-26T21:37:50.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:50.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:50.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:50.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:50.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:50.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:50.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:50.760 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:50.862 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:50.907 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:50.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:51.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:51.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:51.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:51.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.314 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:51.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: pgmap v400: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:51.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: 8.c deep-scrub starts 2024-03-26T21:37:51.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:37:51.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: 8.d scrub starts 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: pgmap v400: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: 8.c deep-scrub starts 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: 8.d scrub starts 2024-03-26T21:37:51.405 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:51 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:51.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:51.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:37:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: pgmap v400: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: 8.c deep-scrub starts 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: 8.d scrub starts 2024-03-26T21:37:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:51 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:37:51.501 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:37:51.501 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:37:51.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:51.595 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:51.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:51.728 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:51.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:51.861 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:51.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:51.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:52.011 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:52.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:52.150 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:52.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.292 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:52.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:52.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 8.d scrub ok 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 8.4 scrub starts 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:52.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 8.d scrub ok 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 8.4 scrub starts 2024-03-26T21:37:52.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:37:52.418 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:37:52.418 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:52.418 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:52.418 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:52 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:52.427 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:52.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 8.d scrub ok 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 8.4 scrub starts 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:37:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:52 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:37:52.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:52.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:52.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:52.696 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:52.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:52.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:52.831 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:52.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:52.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:52.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:53.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:53.094 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:53.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:53.232 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 706 op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 8.4 scrub ok 2024-03-26T21:37:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 706 op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 8.4 scrub ok 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:37:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:53 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:37:53.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:53.366 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:53.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: pgmap v401: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 706 op/s; 3531/168867 objects misplaced (2.091%) 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 8.4 scrub ok 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:37:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:53 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:37:53.503 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:53.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:53.636 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:53.674 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:53.732 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:37:53.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:53.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:53.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:53.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:53.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:54.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:54.042 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:54.128 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:37:54.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:54.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:54.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:54.314 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:54.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:54.449 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3444136026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:54 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:37:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:37:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:37:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:37:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3444136026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3444136026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:54 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:37:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.578 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:54.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:54.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:54.707 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:54.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:54.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:54.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:55.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:55.094 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:55.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.227 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:55.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:55.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: pgmap v402: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:55.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[29406]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: pgmap v402: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:37:55.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:55 smithi055 ceph-mon[40587]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:37:55.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:55.351 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:55.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:55.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: pgmap v402: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:37:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:55 smithi067 ceph-mon[33346]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T21:37:55.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:55.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:55.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:55.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:37:55.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:55.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:37:55.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:56.028 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:56.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:56.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:56.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:56.290 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:56.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:56.422 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:56.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:56.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:56.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:56.697 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:56.798 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:56.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:56.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:56.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:56.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:57.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:57.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:57.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:57.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:57 smithi067 ceph-mon[33346]: pgmap v404: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:57.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:37:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:37:57.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:57 smithi055 ceph-mon[40587]: pgmap v404: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:57.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:57 smithi055 ceph-mon[29406]: pgmap v404: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 3212/155304 objects misplaced (2.068%) 2024-03-26T21:37:57.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:57.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:57.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:57.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:57.608 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:57.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:57.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:57.871 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:57.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:57.973 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:58.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:58.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:58.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:58.274 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:58.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:58.404 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:58 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:37:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:58 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:37:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:58 smithi067 ceph-mon[33346]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:37:58.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:37:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:37:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[40587]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:37:58.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:37:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:37:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:37:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-mon[29406]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T21:37:58.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:58.660 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:58.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:58.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:58.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:37:58.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:58.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:37:59.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:37:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:37:59.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:59.170 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:37:59.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:37:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:37:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:59 smithi055 ceph-mon[40587]: pgmap v406: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 145 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 813 op/s; 3212/149364 objects misplaced (2.150%) 2024-03-26T21:37:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:37:59 smithi055 ceph-mon[40587]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:37:59.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:59 smithi055 ceph-mon[29406]: pgmap v406: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 145 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 813 op/s; 3212/149364 objects misplaced (2.150%) 2024-03-26T21:37:59.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:37:59 smithi055 ceph-mon[29406]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:37:59.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:37:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:37:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:37:59.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:59 smithi067 ceph-mon[33346]: pgmap v406: 148 pgs: 2 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 145 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 813 op/s; 3212/149364 objects misplaced (2.150%) 2024-03-26T21:37:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:37:59 smithi067 ceph-mon[33346]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T21:37:59.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:37:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:37:59.662 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:37:59.695 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:37:59.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:37:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:37:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:37:59.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:37:59.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:37:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:37:59.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:00.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:00.100 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:00.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:00.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:00.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:00 smithi055 ceph-mon[29406]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:38:00.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:00 smithi055 ceph-mon[40587]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:38:00.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:00.364 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:00.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:00 smithi067 ceph-mon[33346]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T21:38:00.499 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:00.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:00.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:00.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:00.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:00.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:00.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:00.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:01.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:01.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:01.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:01.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:01.289 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[40587]: pgmap v409: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 958 B/s rd, 0 B/s wr, 499 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T21:38:01.290 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[40587]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:38:01.290 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:38:01.290 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:38:01.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[29406]: pgmap v409: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 958 B/s rd, 0 B/s wr, 499 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T21:38:01.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[29406]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:38:01.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:38:01.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:01 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:38:01.293 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.350 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:01.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:01.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:01 smithi067 ceph-mon[33346]: pgmap v409: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 958 B/s rd, 0 B/s wr, 499 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T21:38:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:01 smithi067 ceph-mon[33346]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:38:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:01 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:38:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:01 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:38:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.491 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:01.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:01.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:01.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:01.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:01.766 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:01.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:01.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:01.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:02.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:02.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:02.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:02.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:02.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:02.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.426 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:02.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:02 smithi067 ceph-mon[33346]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:38:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:02.496 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:02.496 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[40587]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[29406]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:02.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:02.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:02.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:38:02.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:02.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:02.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:02.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:02.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:02.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:02.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:03.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:03.196 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:38:03.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:03.234 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:03.247 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:38:03.292 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:38:03.336 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:03.340 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:38:03.384 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:38:03.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:03 smithi055 ceph-mon[40587]: pgmap v411: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 267 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T21:38:03.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:03 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1786269967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:03.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:03 smithi055 ceph-mon[29406]: pgmap v411: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 267 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T21:38:03.385 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:03 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1786269967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:03.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:03.433 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:38:03.475 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:38:03.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:03 smithi067 ceph-mon[33346]: pgmap v411: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 267 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T21:38:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:03 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1786269967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:03.523 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:38:03.535 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:03.539 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:38:03.539 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:38:03.540 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:38:03.606 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:38:03.611 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:38:03.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:03.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:03.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:38:03.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:03.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:04.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.084 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:04.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:04.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:04.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:04.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:04.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:04.505 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:04.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:04.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:04.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:04.776 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:04.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:04.907 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:04.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:04.915 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-03-26T21:38:03.464133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2999196096,"num_objects":45954,"num_object_clones":0,"num_object_copies":137862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45954,"num_whiteouts":0,"num_read":1487,"num_read_kb":1333,"num_write":182793,"num_write_kb":7296286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25742,"num_bytes_recovered":1595743404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86808,"ondisk_log_size":86808,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11536680,"kb_used_data":9235724,"kb_used_omap":15,"kb_used_meta":2300784,"kb_avail":738424536,"statfs":{"total":767960285184,"available":756146724864,"internally_reserved":0,"allocated":9457381376,"data_stored":9454056525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":2356003598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-251265024,"num_objects":-3834,"num_object_clones":0,"num_object_copies":-11502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3212,"num_objects_unfound":0,"num_objects_dirty":-3834,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3834,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003688"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.140347+0000","last_change":"2024-03-26T21:37:47.921256+0000","last_active":"2024-03-26T21:37:58.140347+0000","last_peered":"2024-03-26T21:37:58.140347+0000","last_clean":"2024-03-26T21:37:58.140347+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:37:58.140347+0000","last_undegraded":"2024-03-26T21:37:58.140347+0000","last_fullsized":"2024-03-26T21:37:58.140347+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:47.921004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:47.921004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:06.718092+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174708+0000","last_change":"2024-03-26T21:37:51.540963+0000","last_active":"2024-03-26T21:38:01.174708+0000","last_peered":"2024-03-26T21:38:01.174708+0000","last_clean":"2024-03-26T21:38:01.174708+0000","last_became_active":"2024-03-26T21:28:38.866360+0000","last_became_peered":"2024-03-26T21:28:38.866360+0000","last_unstale":"2024-03-26T21:38:01.174708+0000","last_undegraded":"2024-03-26T21:38:01.174708+0000","last_fullsized":"2024-03-26T21:38:01.174708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:51.540837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.540837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.727643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182095+0000","last_change":"2024-03-26T21:35:27.960032+0000","last_active":"2024-03-26T21:38:01.182095+0000","last_peered":"2024-03-26T21:38:01.182095+0000","last_clean":"2024-03-26T21:38:01.182095+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:38:01.182095+0000","last_undegraded":"2024-03-26T21:38:01.182095+0000","last_fullsized":"2024-03-26T21:38:01.182095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.959913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.959913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":61,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.140054+0000","last_change":"2024-03-26T21:36:51.831333+0000","last_active":"2024-03-26T21:37:58.140054+0000","last_peered":"2024-03-26T21:37:58.140054+0000","last_clean":"2024-03-26T21:37:58.140054+0000","last_became_active":"2024-03-26T21:36:51.695519+0000","last_became_peered":"2024-03-26T21:36:51.695519+0000","last_unstale":"2024-03-26T21:37:58.140054+0000","last_undegraded":"2024-03-26T21:37:58.140054+0000","last_fullsized":"2024-03-26T21:37:58.140054+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:35:39.174314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:39.174314+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178544+0000","last_change":"2024-03-26T21:37:53.766654+0000","last_active":"2024-03-26T21:38:01.178544+0000","last_peered":"2024-03-26T21:38:01.178544+0000","last_clean":"2024-03-26T21:38:01.178544+0000","last_became_active":"2024-03-26T21:28:41.117371+0000","last_became_peered":"2024-03-26T21:28:41.117371+0000","last_unstale":"2024-03-26T21:38:01.178544+0000","last_undegraded":"2024-03-26T21:38:01.178544+0000","last_fullsized":"2024-03-26T21:38:01.178544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:53.766318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:37:53.766318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:19.401728+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.139129+0000","last_change":"2024-03-26T21:37:36.866511+0000","last_active":"2024-03-26T21:37:58.139129+0000","last_peered":"2024-03-26T21:37:58.139129+0000","last_clean":"2024-03-26T21:37:58.139129+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:37:58.139129+0000","last_undegraded":"2024-03-26T21:37:58.139129+0000","last_fullsized":"2024-03-26T21:37:58.139129+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:36.866391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:36.866391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:48.905605+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178764+0000","last_change":"2024-03-26T21:37:41.557249+0000","last_active":"2024-03-26T21:38:01.178764+0000","last_peered":"2024-03-26T21:38:01.178764+0000","last_clean":"2024-03-26T21:38:01.178764+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:38:01.178764+0000","last_undegraded":"2024-03-26T21:38:01.178764+0000","last_fullsized":"2024-03-26T21:38:01.178764+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:41.557115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:41.557115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:08.746969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.177432+0000","last_change":"2024-03-26T21:35:24.162334+0000","last_active":"2024-03-26T21:38:01.177432+0000","last_peered":"2024-03-26T21:38:01.177432+0000","last_clean":"2024-03-26T21:38:01.177432+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:38:01.177432+0000","last_undegraded":"2024-03-26T21:38:01.177432+0000","last_fullsized":"2024-03-26T21:38:01.177432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:35:24.162191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:24.162191+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":346,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.180635+0000","last_change":"2024-03-26T21:35:25.946247+0000","last_active":"2024-03-26T21:38:01.180635+0000","last_peered":"2024-03-26T21:38:01.180635+0000","last_clean":"2024-03-26T21:38:01.180635+0000","last_became_active":"2024-03-26T21:28:40.888006+0000","last_became_peered":"2024-03-26T21:28:40.888006+0000","last_unstale":"2024-03-26T21:38:01.180635+0000","last_undegraded":"2024-03-26T21:38:01.180635+0000","last_fullsized":"2024-03-26T21:38:01.180635+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.946125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.946125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171315+0000","last_change":"2024-03-26T21:37:45.305662+0000","last_active":"2024-03-26T21:38:01.171315+0000","last_peered":"2024-03-26T21:38:01.171315+0000","last_clean":"2024-03-26T21:38:01.171315+0000","last_became_active":"2024-03-26T21:28:38.872343+0000","last_became_peered":"2024-03-26T21:28:38.872343+0000","last_unstale":"2024-03-26T21:38:01.171315+0000","last_undegraded":"2024-03-26T21:38:01.171315+0000","last_fullsized":"2024-03-26T21:38:01.171315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:45.305541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:37:45.305541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:09.945938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172435+0000","last_change":"2024-03-26T21:36:51.701112+0000","last_active":"2024-03-26T21:38:01.172435+0000","last_peered":"2024-03-26T21:38:01.172435+0000","last_clean":"2024-03-26T21:38:01.172435+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:38:01.172435+0000","last_undegraded":"2024-03-26T21:38:01.172435+0000","last_fullsized":"2024-03-26T21:38:01.172435+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:35:48.967897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"167'13","reported_seq":69,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.140442+0000","last_change":"2024-03-26T21:37:38.911638+0000","last_active":"2024-03-26T21:37:58.140442+0000","last_peered":"2024-03-26T21:37:58.140442+0000","last_clean":"2024-03-26T21:37:58.140442+0000","last_became_active":"2024-03-26T21:36:51.703479+0000","last_became_peered":"2024-03-26T21:36:51.703479+0000","last_unstale":"2024-03-26T21:37:58.140442+0000","last_undegraded":"2024-03-26T21:37:58.140442+0000","last_fullsized":"2024-03-26T21:37:58.140442+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'13","last_scrub_stamp":"2024-03-26T21:37:38.911399+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:37:38.911399+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:04.223913+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174126+0000","last_change":"2024-03-26T21:37:44.322733+0000","last_active":"2024-03-26T21:38:01.174126+0000","last_peered":"2024-03-26T21:38:01.174126+0000","last_clean":"2024-03-26T21:38:01.174126+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:38:01.174126+0000","last_undegraded":"2024-03-26T21:38:01.174126+0000","last_fullsized":"2024-03-26T21:38:01.174126+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:44.322504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:37:44.322504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:53.719827+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":89,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182373+0000","last_change":"2024-03-26T21:37:28.013192+0000","last_active":"2024-03-26T21:38:01.182373+0000","last_peered":"2024-03-26T21:38:01.182373+0000","last_clean":"2024-03-26T21:38:01.182373+0000","last_became_active":"2024-03-26T21:36:51.679940+0000","last_became_peered":"2024-03-26T21:36:51.679940+0000","last_unstale":"2024-03-26T21:38:01.182373+0000","last_undegraded":"2024-03-26T21:38:01.182373+0000","last_fullsized":"2024-03-26T21:38:01.182373+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:28.013110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:28.013110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:56.631916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172825+0000","last_change":"2024-03-26T21:36:51.677654+0000","last_active":"2024-03-26T21:38:01.172825+0000","last_peered":"2024-03-26T21:38:01.172825+0000","last_clean":"2024-03-26T21:38:01.172825+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:38:01.172825+0000","last_undegraded":"2024-03-26T21:38:01.172825+0000","last_fullsized":"2024-03-26T21:38:01.172825+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:39.913327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:39.913327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188343+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":57,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.142083+0000","last_change":"2024-03-26T21:36:51.841108+0000","last_active":"2024-03-26T21:37:58.142083+0000","last_peered":"2024-03-26T21:37:58.142083+0000","last_clean":"2024-03-26T21:37:58.142083+0000","last_became_active":"2024-03-26T21:36:51.700845+0000","last_became_peered":"2024-03-26T21:36:51.700845+0000","last_unstale":"2024-03-26T21:37:58.142083+0000","last_undegraded":"2024-03-26T21:37:58.142083+0000","last_fullsized":"2024-03-26T21:37:58.142083+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:35:36.208704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:36.208704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141884+0000","last_change":"2024-03-26T21:36:51.689429+0000","last_active":"2024-03-26T21:37:58.141884+0000","last_peered":"2024-03-26T21:37:58.141884+0000","last_clean":"2024-03-26T21:37:58.141884+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:37:58.141884+0000","last_undegraded":"2024-03-26T21:37:58.141884+0000","last_fullsized":"2024-03-26T21:37:58.141884+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:38.197384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.197384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174509+0000","last_change":"2024-03-26T21:37:23.346662+0000","last_active":"2024-03-26T21:38:01.174509+0000","last_peered":"2024-03-26T21:38:01.174509+0000","last_clean":"2024-03-26T21:38:01.174509+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:38:01.174509+0000","last_undegraded":"2024-03-26T21:38:01.174509+0000","last_fullsized":"2024-03-26T21:38:01.174509+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:23.346550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:23.346550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:28.303628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181515+0000","last_change":"2024-03-26T21:37:42.479993+0000","last_active":"2024-03-26T21:38:01.181515+0000","last_peered":"2024-03-26T21:38:01.181515+0000","last_clean":"2024-03-26T21:38:01.181515+0000","last_became_active":"2024-03-26T21:36:51.679640+0000","last_became_peered":"2024-03-26T21:36:51.679640+0000","last_unstale":"2024-03-26T21:38:01.181515+0000","last_undegraded":"2024-03-26T21:38:01.181515+0000","last_fullsized":"2024-03-26T21:38:01.181515+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:42.479878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:42.479878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:55.417791+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":395,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183802+0000","last_change":"2024-03-26T21:37:32.810165+0000","last_active":"2024-03-26T21:38:01.183802+0000","last_peered":"2024-03-26T21:38:01.183802+0000","last_clean":"2024-03-26T21:38:01.183802+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:38:01.183802+0000","last_undegraded":"2024-03-26T21:38:01.183802+0000","last_fullsized":"2024-03-26T21:38:01.183802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:37:32.810013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:32.810013+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:44.336549+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183118+0000","last_change":"2024-03-26T21:35:27.311601+0000","last_active":"2024-03-26T21:38:01.183118+0000","last_peered":"2024-03-26T21:38:01.183118+0000","last_clean":"2024-03-26T21:38:01.183118+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:38:01.183118+0000","last_undegraded":"2024-03-26T21:38:01.183118+0000","last_fullsized":"2024-03-26T21:38:01.183118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.311473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.311473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735288+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":59,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141734+0000","last_change":"2024-03-26T21:37:33.861905+0000","last_active":"2024-03-26T21:37:58.141734+0000","last_peered":"2024-03-26T21:37:58.141734+0000","last_clean":"2024-03-26T21:37:58.141734+0000","last_became_active":"2024-03-26T21:36:51.700623+0000","last_became_peered":"2024-03-26T21:36:51.700623+0000","last_unstale":"2024-03-26T21:37:58.141734+0000","last_undegraded":"2024-03-26T21:37:58.141734+0000","last_fullsized":"2024-03-26T21:37:58.141734+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:33.861808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:33.861808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:59.138721+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.140313+0000","last_change":"2024-03-26T21:36:51.708559+0000","last_active":"2024-03-26T21:37:58.140313+0000","last_peered":"2024-03-26T21:37:58.140313+0000","last_clean":"2024-03-26T21:37:58.140313+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:37:58.140313+0000","last_undegraded":"2024-03-26T21:37:58.140313+0000","last_fullsized":"2024-03-26T21:37:58.140313+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.095921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.095921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.742995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":414,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178431+0000","last_change":"2024-03-26T21:36:51.825498+0000","last_active":"2024-03-26T21:38:01.178431+0000","last_peered":"2024-03-26T21:38:01.178431+0000","last_clean":"2024-03-26T21:38:01.178431+0000","last_became_active":"2024-03-26T21:36:51.690591+0000","last_became_peered":"2024-03-26T21:36:51.690591+0000","last_unstale":"2024-03-26T21:38:01.178431+0000","last_undegraded":"2024-03-26T21:38:01.178431+0000","last_fullsized":"2024-03-26T21:38:01.178431+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:36:42.463060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:36:42.463060+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:10.408183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183498+0000","last_change":"2024-03-26T21:35:49.245930+0000","last_active":"2024-03-26T21:38:01.183498+0000","last_peered":"2024-03-26T21:38:01.183498+0000","last_clean":"2024-03-26T21:38:01.183498+0000","last_became_active":"2024-03-26T21:28:40.884093+0000","last_became_peered":"2024-03-26T21:28:40.884093+0000","last_unstale":"2024-03-26T21:38:01.183498+0000","last_undegraded":"2024-03-26T21:38:01.183498+0000","last_fullsized":"2024-03-26T21:38:01.183498+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:35:49.245837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:49.245837+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"205'12","reported_seq":516,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.176241+0000","last_change":"2024-03-26T21:36:51.750736+0000","last_active":"2024-03-26T21:38:01.176241+0000","last_peered":"2024-03-26T21:38:01.176241+0000","last_clean":"2024-03-26T21:38:01.176241+0000","last_became_active":"2024-03-26T21:36:51.686113+0000","last_became_peered":"2024-03-26T21:36:51.686113+0000","last_unstale":"2024-03-26T21:38:01.176241+0000","last_undegraded":"2024-03-26T21:38:01.176241+0000","last_fullsized":"2024-03-26T21:38:01.176241+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10","last_scrub_stamp":"2024-03-26T21:35:48.067927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:48.067927+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":397,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174922+0000","last_change":"2024-03-26T21:37:19.291308+0000","last_active":"2024-03-26T21:38:01.174922+0000","last_peered":"2024-03-26T21:38:01.174922+0000","last_clean":"2024-03-26T21:38:01.174922+0000","last_became_active":"2024-03-26T21:36:51.709840+0000","last_became_peered":"2024-03-26T21:36:51.709840+0000","last_unstale":"2024-03-26T21:38:01.174922+0000","last_undegraded":"2024-03-26T21:38:01.174922+0000","last_fullsized":"2024-03-26T21:38:01.174922+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:37:19.291181+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:37:19.291181+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:42.469366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":370,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.187749+0000","last_change":"2024-03-26T21:36:51.237434+0000","last_active":"2024-03-26T21:38:01.187749+0000","last_peered":"2024-03-26T21:38:01.187749+0000","last_clean":"2024-03-26T21:38:01.187749+0000","last_became_active":"2024-03-26T21:28:37.141360+0000","last_became_peered":"2024-03-26T21:28:37.141360+0000","last_unstale":"2024-03-26T21:38:01.187749+0000","last_undegraded":"2024-03-26T21:38:01.187749+0000","last_fullsized":"2024-03-26T21:38:01.187749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:36:51.237303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:36:51.237303+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:19.199155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174260+0000","last_change":"2024-03-26T21:36:51.713647+0000","last_active":"2024-03-26T21:38:01.174260+0000","last_peered":"2024-03-26T21:38:01.174260+0000","last_clean":"2024-03-26T21:38:01.174260+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:38:01.174260+0000","last_undegraded":"2024-03-26T21:38:01.174260+0000","last_fullsized":"2024-03-26T21:38:01.174260+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:36:46.389418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:36:46.389418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.188289+0000","last_change":"2024-03-26T21:37:51.782264+0000","last_active":"2024-03-26T21:38:01.188289+0000","last_peered":"2024-03-26T21:38:01.188289+0000","last_clean":"2024-03-26T21:38:01.188289+0000","last_became_active":"2024-03-26T21:28:38.866999+0000","last_became_peered":"2024-03-26T21:28:38.866999+0000","last_unstale":"2024-03-26T21:38:01.188289+0000","last_undegraded":"2024-03-26T21:38:01.188289+0000","last_fullsized":"2024-03-26T21:38:01.188289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:51.775881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.775881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:02.292448+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.177972+0000","last_change":"2024-03-26T21:35:20.257839+0000","last_active":"2024-03-26T21:38:01.177972+0000","last_peered":"2024-03-26T21:38:01.177972+0000","last_clean":"2024-03-26T21:38:01.177972+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:38:01.177972+0000","last_undegraded":"2024-03-26T21:38:01.177972+0000","last_fullsized":"2024-03-26T21:38:01.177972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:20.257696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:20.257696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"167'16","reported_seq":395,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.176137+0000","last_change":"2024-03-26T21:35:29.097610+0000","last_active":"2024-03-26T21:38:01.176137+0000","last_peered":"2024-03-26T21:38:01.176137+0000","last_clean":"2024-03-26T21:38:01.176137+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:38:01.176137+0000","last_undegraded":"2024-03-26T21:38:01.176137+0000","last_fullsized":"2024-03-26T21:38:01.176137+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'16","last_scrub_stamp":"2024-03-26T21:35:29.097470+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:35:29.097470+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":368,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182614+0000","last_change":"2024-03-26T21:36:51.681374+0000","last_active":"2024-03-26T21:38:01.182614+0000","last_peered":"2024-03-26T21:38:01.182614+0000","last_clean":"2024-03-26T21:38:01.182614+0000","last_became_active":"2024-03-26T21:36:51.681091+0000","last_became_peered":"2024-03-26T21:36:51.681091+0000","last_unstale":"2024-03-26T21:38:01.182614+0000","last_undegraded":"2024-03-26T21:38:01.182614+0000","last_fullsized":"2024-03-26T21:38:01.182614+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:37.291480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:37.291480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.175630+0000","last_change":"2024-03-26T21:36:51.685996+0000","last_active":"2024-03-26T21:38:01.175630+0000","last_peered":"2024-03-26T21:38:01.175630+0000","last_clean":"2024-03-26T21:38:01.175630+0000","last_became_active":"2024-03-26T21:36:51.685541+0000","last_became_peered":"2024-03-26T21:36:51.685541+0000","last_unstale":"2024-03-26T21:38:01.175630+0000","last_undegraded":"2024-03-26T21:38:01.175630+0000","last_fullsized":"2024-03-26T21:38:01.175630+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:35:43.135432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182292+0000","last_change":"2024-03-26T21:35:33.934130+0000","last_active":"2024-03-26T21:38:01.182292+0000","last_peered":"2024-03-26T21:38:01.182292+0000","last_clean":"2024-03-26T21:38:01.182292+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:38:01.182292+0000","last_undegraded":"2024-03-26T21:38:01.182292+0000","last_fullsized":"2024-03-26T21:38:01.182292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:33.934014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.934014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178107+0000","last_change":"2024-03-26T21:37:48.600196+0000","last_active":"2024-03-26T21:38:01.178107+0000","last_peered":"2024-03-26T21:38:01.178107+0000","last_clean":"2024-03-26T21:38:01.178107+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:38:01.178107+0000","last_undegraded":"2024-03-26T21:38:01.178107+0000","last_fullsized":"2024-03-26T21:38:01.178107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:37:48.600034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:48.600034+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:11.987318+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178766+0000","last_change":"2024-03-26T21:35:48.403013+0000","last_active":"2024-03-26T21:38:01.178766+0000","last_peered":"2024-03-26T21:38:01.178766+0000","last_clean":"2024-03-26T21:38:01.178766+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:38:01.178766+0000","last_undegraded":"2024-03-26T21:38:01.178766+0000","last_fullsized":"2024-03-26T21:38:01.178766+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:48.402059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:35:48.402059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":364,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182007+0000","last_change":"2024-03-26T21:36:51.677483+0000","last_active":"2024-03-26T21:38:01.182007+0000","last_peered":"2024-03-26T21:38:01.182007+0000","last_clean":"2024-03-26T21:38:01.182007+0000","last_became_active":"2024-03-26T21:36:51.677119+0000","last_became_peered":"2024-03-26T21:36:51.677119+0000","last_unstale":"2024-03-26T21:38:01.182007+0000","last_undegraded":"2024-03-26T21:38:01.182007+0000","last_fullsized":"2024-03-26T21:38:01.182007+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.435219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.435219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171626+0000","last_change":"2024-03-26T21:37:24.232769+0000","last_active":"2024-03-26T21:38:01.171626+0000","last_peered":"2024-03-26T21:38:01.171626+0000","last_clean":"2024-03-26T21:38:01.171626+0000","last_became_active":"2024-03-26T21:28:34.847677+0000","last_became_peered":"2024-03-26T21:28:34.847677+0000","last_unstale":"2024-03-26T21:38:01.171626+0000","last_undegraded":"2024-03-26T21:38:01.171626+0000","last_fullsized":"2024-03-26T21:38:01.171626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:24.232664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:24.232664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:43.004607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":374,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.177380+0000","last_change":"2024-03-26T21:35:22.254470+0000","last_active":"2024-03-26T21:38:01.177380+0000","last_peered":"2024-03-26T21:38:01.177380+0000","last_clean":"2024-03-26T21:38:01.177380+0000","last_became_active":"2024-03-26T21:28:37.142339+0000","last_became_peered":"2024-03-26T21:28:37.142339+0000","last_unstale":"2024-03-26T21:38:01.177380+0000","last_undegraded":"2024-03-26T21:38:01.177380+0000","last_fullsized":"2024-03-26T21:38:01.177380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:35:22.254300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:22.254300+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.180353+0000","last_change":"2024-03-26T21:35:49.844007+0000","last_active":"2024-03-26T21:38:01.180353+0000","last_peered":"2024-03-26T21:38:01.180353+0000","last_clean":"2024-03-26T21:38:01.180353+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:38:01.180353+0000","last_undegraded":"2024-03-26T21:38:01.180353+0000","last_fullsized":"2024-03-26T21:38:01.180353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:49.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:49.843892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:06.609041+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178492+0000","last_change":"2024-03-26T21:35:19.231094+0000","last_active":"2024-03-26T21:38:01.178492+0000","last_peered":"2024-03-26T21:38:01.178492+0000","last_clean":"2024-03-26T21:38:01.178492+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:38:01.178492+0000","last_undegraded":"2024-03-26T21:38:01.178492+0000","last_fullsized":"2024-03-26T21:38:01.178492+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.230859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.230859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":379,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182070+0000","last_change":"2024-03-26T21:36:51.682314+0000","last_active":"2024-03-26T21:38:01.182070+0000","last_peered":"2024-03-26T21:38:01.182070+0000","last_clean":"2024-03-26T21:38:01.182070+0000","last_became_active":"2024-03-26T21:36:51.681971+0000","last_became_peered":"2024-03-26T21:36:51.681971+0000","last_unstale":"2024-03-26T21:38:01.182070+0000","last_undegraded":"2024-03-26T21:38:01.182070+0000","last_fullsized":"2024-03-26T21:38:01.182070+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:26.578385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:26.578385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"91'13","reported_seq":391,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.179159+0000","last_change":"2024-03-26T21:37:43.413704+0000","last_active":"2024-03-26T21:38:01.179159+0000","last_peered":"2024-03-26T21:38:01.179159+0000","last_clean":"2024-03-26T21:38:01.179159+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:38:01.179159+0000","last_undegraded":"2024-03-26T21:38:01.179159+0000","last_fullsized":"2024-03-26T21:38:01.179159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'13","last_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:37:43.412986+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:12.965507+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.177071+0000","last_change":"2024-03-26T21:35:13.297009+0000","last_active":"2024-03-26T21:38:01.177071+0000","last_peered":"2024-03-26T21:38:01.177071+0000","last_clean":"2024-03-26T21:38:01.177071+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:38:01.177071+0000","last_undegraded":"2024-03-26T21:38:01.177071+0000","last_fullsized":"2024-03-26T21:38:01.177071+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.296881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.296881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178396+0000","last_change":"2024-03-26T21:37:36.535939+0000","last_active":"2024-03-26T21:38:01.178396+0000","last_peered":"2024-03-26T21:38:01.178396+0000","last_clean":"2024-03-26T21:38:01.178396+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:38:01.178396+0000","last_undegraded":"2024-03-26T21:38:01.178396+0000","last_fullsized":"2024-03-26T21:38:01.178396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:36.535803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:36.535803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:49.490096+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141406+0000","last_change":"2024-03-26T21:36:51.701180+0000","last_active":"2024-03-26T21:37:58.141406+0000","last_peered":"2024-03-26T21:37:58.141406+0000","last_clean":"2024-03-26T21:37:58.141406+0000","last_became_active":"2024-03-26T21:36:51.692968+0000","last_became_peered":"2024-03-26T21:36:51.692968+0000","last_unstale":"2024-03-26T21:37:58.141406+0000","last_undegraded":"2024-03-26T21:37:58.141406+0000","last_fullsized":"2024-03-26T21:37:58.141406+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:28.148936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.148936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":71,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141369+0000","last_change":"2024-03-26T21:37:32.834549+0000","last_active":"2024-03-26T21:37:58.141369+0000","last_peered":"2024-03-26T21:37:58.141369+0000","last_clean":"2024-03-26T21:37:58.141369+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:37:58.141369+0000","last_undegraded":"2024-03-26T21:37:58.141369+0000","last_fullsized":"2024-03-26T21:37:58.141369+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:37:32.834417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:32.834417+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:38.231285+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.187431+0000","last_change":"2024-03-26T21:35:11.986871+0000","last_active":"2024-03-26T21:38:01.187431+0000","last_peered":"2024-03-26T21:38:01.187431+0000","last_clean":"2024-03-26T21:38:01.187431+0000","last_became_active":"2024-03-26T21:28:40.891130+0000","last_became_peered":"2024-03-26T21:28:40.891130+0000","last_unstale":"2024-03-26T21:38:01.187431+0000","last_undegraded":"2024-03-26T21:38:01.187431+0000","last_fullsized":"2024-03-26T21:38:01.187431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:35:11.986741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.986741+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":481,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172525+0000","last_change":"2024-03-26T21:37:37.195234+0000","last_active":"2024-03-26T21:38:01.172525+0000","last_peered":"2024-03-26T21:38:01.172525+0000","last_clean":"2024-03-26T21:38:01.172525+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:38:01.172525+0000","last_undegraded":"2024-03-26T21:38:01.172525+0000","last_fullsized":"2024-03-26T21:38:01.172525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:37:37.194929+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:37:37.194929+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.615122+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182211+0000","last_change":"2024-03-26T21:36:51.677366+0000","last_active":"2024-03-26T21:38:01.182211+0000","last_peered":"2024-03-26T21:38:01.182211+0000","last_clean":"2024-03-26T21:38:01.182211+0000","last_became_active":"2024-03-26T21:36:51.677035+0000","last_became_peered":"2024-03-26T21:36:51.677035+0000","last_unstale":"2024-03-26T21:38:01.182211+0000","last_undegraded":"2024-03-26T21:38:01.182211+0000","last_fullsized":"2024-03-26T21:38:01.182211+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.470873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.470873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"166'15","reported_seq":383,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182994+0000","last_change":"2024-03-26T21:37:38.529390+0000","last_active":"2024-03-26T21:38:01.182994+0000","last_peered":"2024-03-26T21:38:01.182994+0000","last_clean":"2024-03-26T21:38:01.182994+0000","last_became_active":"2024-03-26T21:28:37.140637+0000","last_became_peered":"2024-03-26T21:28:37.140637+0000","last_unstale":"2024-03-26T21:38:01.182994+0000","last_undegraded":"2024-03-26T21:38:01.182994+0000","last_fullsized":"2024-03-26T21:38:01.182994+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:37:38.528979+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:37:38.528979+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.509119+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.142172+0000","last_change":"2024-03-26T21:36:51.674400+0000","last_active":"2024-03-26T21:37:58.142172+0000","last_peered":"2024-03-26T21:37:58.142172+0000","last_clean":"2024-03-26T21:37:58.142172+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:37:58.142172+0000","last_undegraded":"2024-03-26T21:37:58.142172+0000","last_fullsized":"2024-03-26T21:37:58.142172+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:31.255065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:31.255065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":476,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183203+0000","last_change":"2024-03-26T21:35:40.415843+0000","last_active":"2024-03-26T21:38:01.183203+0000","last_peered":"2024-03-26T21:38:01.183203+0000","last_clean":"2024-03-26T21:38:01.183203+0000","last_became_active":"2024-03-26T21:28:38.864277+0000","last_became_peered":"2024-03-26T21:28:38.864277+0000","last_unstale":"2024-03-26T21:38:01.183203+0000","last_undegraded":"2024-03-26T21:38:01.183203+0000","last_fullsized":"2024-03-26T21:38:01.183203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:35:40.415587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:40.415587+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181854+0000","last_change":"2024-03-26T21:35:31.926557+0000","last_active":"2024-03-26T21:38:01.181854+0000","last_peered":"2024-03-26T21:38:01.181854+0000","last_clean":"2024-03-26T21:38:01.181854+0000","last_became_active":"2024-03-26T21:28:34.840764+0000","last_became_peered":"2024-03-26T21:28:34.840764+0000","last_unstale":"2024-03-26T21:38:01.181854+0000","last_undegraded":"2024-03-26T21:38:01.181854+0000","last_fullsized":"2024-03-26T21:38:01.181854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:31.926437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:31.926437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"146'13","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172760+0000","last_change":"2024-03-26T21:35:09.889342+0000","last_active":"2024-03-26T21:38:01.172760+0000","last_peered":"2024-03-26T21:38:01.172760+0000","last_clean":"2024-03-26T21:38:01.172760+0000","last_became_active":"2024-03-26T21:28:37.141699+0000","last_became_peered":"2024-03-26T21:28:37.141699+0000","last_unstale":"2024-03-26T21:38:01.172760+0000","last_undegraded":"2024-03-26T21:38:01.172760+0000","last_fullsized":"2024-03-26T21:38:01.172760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:35:09.889159+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:35:09.889159+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188260+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.184428+0000","last_change":"2024-03-26T21:37:48.703429+0000","last_active":"2024-03-26T21:38:01.184428+0000","last_peered":"2024-03-26T21:38:01.184428+0000","last_clean":"2024-03-26T21:38:01.184428+0000","last_became_active":"2024-03-26T21:36:51.709349+0000","last_became_peered":"2024-03-26T21:36:51.709349+0000","last_unstale":"2024-03-26T21:38:01.184428+0000","last_undegraded":"2024-03-26T21:38:01.184428+0000","last_fullsized":"2024-03-26T21:38:01.184428+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:48.703325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:48.703325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:48.924651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183592+0000","last_change":"2024-03-26T21:35:36.335634+0000","last_active":"2024-03-26T21:38:01.183592+0000","last_peered":"2024-03-26T21:38:01.183592+0000","last_clean":"2024-03-26T21:38:01.183592+0000","last_became_active":"2024-03-26T21:28:38.863991+0000","last_became_peered":"2024-03-26T21:28:38.863991+0000","last_unstale":"2024-03-26T21:38:01.183592+0000","last_undegraded":"2024-03-26T21:38:01.183592+0000","last_fullsized":"2024-03-26T21:38:01.183592+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:36.335509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:36.335509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735342+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181960+0000","last_change":"2024-03-26T21:36:51.672853+0000","last_active":"2024-03-26T21:38:01.181960+0000","last_peered":"2024-03-26T21:38:01.181960+0000","last_clean":"2024-03-26T21:38:01.181960+0000","last_became_active":"2024-03-26T21:36:51.672500+0000","last_became_peered":"2024-03-26T21:36:51.672500+0000","last_unstale":"2024-03-26T21:38:01.181960+0000","last_undegraded":"2024-03-26T21:38:01.181960+0000","last_fullsized":"2024-03-26T21:38:01.181960+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:21.539821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:21.539821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"113'13","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.187848+0000","last_change":"2024-03-26T21:37:51.120255+0000","last_active":"2024-03-26T21:38:01.187848+0000","last_peered":"2024-03-26T21:38:01.187848+0000","last_clean":"2024-03-26T21:38:01.187848+0000","last_became_active":"2024-03-26T21:28:37.141310+0000","last_became_peered":"2024-03-26T21:28:37.141310+0000","last_unstale":"2024-03-26T21:38:01.187848+0000","last_undegraded":"2024-03-26T21:38:01.187848+0000","last_fullsized":"2024-03-26T21:38:01.187848+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:37:51.120100+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.120100+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:21.064739+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181890+0000","last_change":"2024-03-26T21:36:51.697394+0000","last_active":"2024-03-26T21:38:01.181890+0000","last_peered":"2024-03-26T21:38:01.181890+0000","last_clean":"2024-03-26T21:38:01.181890+0000","last_became_active":"2024-03-26T21:36:51.696817+0000","last_became_peered":"2024-03-26T21:36:51.696817+0000","last_unstale":"2024-03-26T21:38:01.181890+0000","last_undegraded":"2024-03-26T21:38:01.181890+0000","last_fullsized":"2024-03-26T21:38:01.181890+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:38.290837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.290837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171855+0000","last_change":"2024-03-26T21:35:30.820470+0000","last_active":"2024-03-26T21:38:01.171855+0000","last_peered":"2024-03-26T21:38:01.171855+0000","last_clean":"2024-03-26T21:38:01.171855+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:38:01.171855+0000","last_undegraded":"2024-03-26T21:38:01.171855+0000","last_fullsized":"2024-03-26T21:38:01.171855+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:30.820356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:30.820356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.175049+0000","last_change":"2024-03-26T21:35:12.106142+0000","last_active":"2024-03-26T21:38:01.175049+0000","last_peered":"2024-03-26T21:38:01.175049+0000","last_clean":"2024-03-26T21:38:01.175049+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:38:01.175049+0000","last_undegraded":"2024-03-26T21:38:01.175049+0000","last_fullsized":"2024-03-26T21:38:01.175049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:12.106061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:12.106061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":377,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183432+0000","last_change":"2024-03-26T21:35:38.322099+0000","last_active":"2024-03-26T21:38:01.183432+0000","last_peered":"2024-03-26T21:38:01.183432+0000","last_clean":"2024-03-26T21:38:01.183432+0000","last_became_active":"2024-03-26T21:28:59.680681+0000","last_became_peered":"2024-03-26T21:28:59.680681+0000","last_unstale":"2024-03-26T21:38:01.183432+0000","last_undegraded":"2024-03-26T21:38:01.183432+0000","last_fullsized":"2024-03-26T21:38:01.183432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:35:38.321844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.321844+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735316+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"227'11298","reported_seq":22004,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.506286+0000","last_change":"2024-03-26T21:37:52.000361+0000","last_active":"2024-03-26T21:38:01.506286+0000","last_peered":"2024-03-26T21:38:01.506286+0000","last_clean":"2024-03-26T21:38:01.506286+0000","last_became_active":"2024-03-26T21:37:23.696825+0000","last_became_peered":"2024-03-26T21:37:23.696825+0000","last_unstale":"2024-03-26T21:38:01.506286+0000","last_undegraded":"2024-03-26T21:38:01.506286+0000","last_fullsized":"2024-03-26T21:38:01.506286+0000","mapping_epoch":213,"log_start":"192'6016","ondisk_log_start":"192'6016","created":173,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10961","last_scrub_stamp":"2024-03-26T21:37:52.000120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:52.000120+0000","objects_scrubbed":3221,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:16.917431+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187826176,"num_objects":2866,"num_object_clones":0,"num_object_copies":8598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5264,"num_bytes_recovered":316866560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171420+0000","last_change":"2024-03-26T21:35:19.938802+0000","last_active":"2024-03-26T21:38:01.171420+0000","last_peered":"2024-03-26T21:38:01.171420+0000","last_clean":"2024-03-26T21:38:01.171420+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:38:01.171420+0000","last_undegraded":"2024-03-26T21:38:01.171420+0000","last_fullsized":"2024-03-26T21:38:01.171420+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:19.938637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:19.938637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188226+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":414,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182788+0000","last_change":"2024-03-26T21:37:32.137328+0000","last_active":"2024-03-26T21:38:01.182788+0000","last_peered":"2024-03-26T21:38:01.182788+0000","last_clean":"2024-03-26T21:38:01.182788+0000","last_became_active":"2024-03-26T21:36:51.680685+0000","last_became_peered":"2024-03-26T21:36:51.680685+0000","last_unstale":"2024-03-26T21:38:01.182788+0000","last_undegraded":"2024-03-26T21:38:01.182788+0000","last_fullsized":"2024-03-26T21:38:01.182788+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:37:32.137190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:32.137190+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:35.662210+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":388,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171199+0000","last_change":"2024-03-26T21:37:46.331357+0000","last_active":"2024-03-26T21:38:01.171199+0000","last_peered":"2024-03-26T21:38:01.171199+0000","last_clean":"2024-03-26T21:38:01.171199+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:38:01.171199+0000","last_undegraded":"2024-03-26T21:38:01.171199+0000","last_fullsized":"2024-03-26T21:38:01.171199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:37:46.331230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:46.331230+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:52.061184+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"205'12","reported_seq":505,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182216+0000","last_change":"2024-03-26T21:37:29.052918+0000","last_active":"2024-03-26T21:38:01.182216+0000","last_peered":"2024-03-26T21:38:01.182216+0000","last_clean":"2024-03-26T21:38:01.182216+0000","last_became_active":"2024-03-26T21:36:51.672126+0000","last_became_peered":"2024-03-26T21:36:51.672126+0000","last_unstale":"2024-03-26T21:38:01.182216+0000","last_undegraded":"2024-03-26T21:38:01.182216+0000","last_fullsized":"2024-03-26T21:38:01.182216+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:37:29.052778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:29.052778+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:45.716698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"227'11219","reported_seq":19507,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.562181+0000","last_change":"2024-03-26T21:37:36.938127+0000","last_active":"2024-03-26T21:38:01.562181+0000","last_peered":"2024-03-26T21:38:01.562181+0000","last_clean":"2024-03-26T21:38:01.562181+0000","last_became_active":"2024-03-26T21:37:22.672219+0000","last_became_peered":"2024-03-26T21:37:22.672219+0000","last_unstale":"2024-03-26T21:38:01.562181+0000","last_undegraded":"2024-03-26T21:38:01.562181+0000","last_fullsized":"2024-03-26T21:38:01.562181+0000","mapping_epoch":212,"log_start":"189'5000","ondisk_log_start":"189'5000","created":173,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'9888","last_scrub_stamp":"2024-03-26T21:37:36.937983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:36.937983+0000","objects_scrubbed":4203,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:53.693240+0000","scrub_duration":1100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186318848,"num_objects":2843,"num_object_clones":0,"num_object_copies":8529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4051,"num_bytes_recovered":237043712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178636+0000","last_change":"2024-03-26T21:35:45.377181+0000","last_active":"2024-03-26T21:38:01.178636+0000","last_peered":"2024-03-26T21:38:01.178636+0000","last_clean":"2024-03-26T21:38:01.178636+0000","last_became_active":"2024-03-26T21:28:40.888610+0000","last_became_peered":"2024-03-26T21:28:40.888610+0000","last_unstale":"2024-03-26T21:38:01.178636+0000","last_undegraded":"2024-03-26T21:38:01.178636+0000","last_fullsized":"2024-03-26T21:38:01.178636+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:45.377075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:45.377075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735367+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":403,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182286+0000","last_change":"2024-03-26T21:37:47.046330+0000","last_active":"2024-03-26T21:38:01.182286+0000","last_peered":"2024-03-26T21:38:01.182286+0000","last_clean":"2024-03-26T21:38:01.182286+0000","last_became_active":"2024-03-26T21:36:51.680045+0000","last_became_peered":"2024-03-26T21:36:51.680045+0000","last_unstale":"2024-03-26T21:38:01.182286+0000","last_undegraded":"2024-03-26T21:38:01.182286+0000","last_fullsized":"2024-03-26T21:38:01.182286+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:37:47.046179+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:37:47.046179+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:13.862762+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":63,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.139791+0000","last_change":"2024-03-26T21:36:51.825893+0000","last_active":"2024-03-26T21:37:58.139791+0000","last_peered":"2024-03-26T21:37:58.139791+0000","last_clean":"2024-03-26T21:37:58.139791+0000","last_became_active":"2024-03-26T21:36:51.690121+0000","last_became_peered":"2024-03-26T21:36:51.690121+0000","last_unstale":"2024-03-26T21:37:58.139791+0000","last_undegraded":"2024-03-26T21:37:58.139791+0000","last_fullsized":"2024-03-26T21:37:58.139791+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:36:22.791543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:36:22.791543+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743229+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.180915+0000","last_change":"2024-03-26T21:37:46.668555+0000","last_active":"2024-03-26T21:38:01.180915+0000","last_peered":"2024-03-26T21:38:01.180915+0000","last_clean":"2024-03-26T21:38:01.180915+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:38:01.180915+0000","last_undegraded":"2024-03-26T21:38:01.180915+0000","last_fullsized":"2024-03-26T21:38:01.180915+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:46.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:46.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:15.710473+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"227'13701","reported_seq":13917,"reported_epoch":227,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-26T21:38:03.148632+0000","last_change":"2024-03-26T21:37:48.570320+0000","last_active":"2024-03-26T21:38:03.148632+0000","last_peered":"2024-03-26T21:38:03.148632+0000","last_clean":"2024-03-26T21:38:03.148632+0000","last_became_active":"2024-03-26T21:34:59.371417+0000","last_became_peered":"2024-03-26T21:34:59.371417+0000","last_unstale":"2024-03-26T21:38:03.148632+0000","last_undegraded":"2024-03-26T21:38:03.148632+0000","last_fullsized":"2024-03-26T21:38:03.148632+0000","mapping_epoch":173,"log_start":"206'7844","ondisk_log_start":"206'7844","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":2324,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180027392,"num_objects":2747,"num_object_clones":0,"num_object_copies":8241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":451968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181667+0000","last_change":"2024-03-26T21:37:24.957358+0000","last_active":"2024-03-26T21:38:01.181667+0000","last_peered":"2024-03-26T21:38:01.181667+0000","last_clean":"2024-03-26T21:38:01.181667+0000","last_became_active":"2024-03-26T21:28:40.886603+0000","last_became_peered":"2024-03-26T21:28:40.886603+0000","last_unstale":"2024-03-26T21:38:01.181667+0000","last_undegraded":"2024-03-26T21:38:01.181667+0000","last_fullsized":"2024-03-26T21:38:01.181667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:24.957265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:24.957265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:25.345994+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.184479+0000","last_change":"2024-03-26T21:37:39.697606+0000","last_active":"2024-03-26T21:38:01.184479+0000","last_peered":"2024-03-26T21:38:01.184479+0000","last_clean":"2024-03-26T21:38:01.184479+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:38:01.184479+0000","last_undegraded":"2024-03-26T21:38:01.184479+0000","last_fullsized":"2024-03-26T21:38:01.184479+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:39.697494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:39.697494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.103753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182290+0000","last_change":"2024-03-26T21:37:41.026493+0000","last_active":"2024-03-26T21:38:01.182290+0000","last_peered":"2024-03-26T21:38:01.182290+0000","last_clean":"2024-03-26T21:38:01.182290+0000","last_became_active":"2024-03-26T21:28:34.844984+0000","last_became_peered":"2024-03-26T21:28:34.844984+0000","last_unstale":"2024-03-26T21:38:01.182290+0000","last_undegraded":"2024-03-26T21:38:01.182290+0000","last_fullsized":"2024-03-26T21:38:01.182290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:41.026360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:41.026360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:43.001315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"110'28","reported_seq":411,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.175529+0000","last_change":"2024-03-26T21:37:33.889201+0000","last_active":"2024-03-26T21:38:01.175529+0000","last_peered":"2024-03-26T21:38:01.175529+0000","last_clean":"2024-03-26T21:38:01.175529+0000","last_became_active":"2024-03-26T21:28:37.141946+0000","last_became_peered":"2024-03-26T21:28:37.141946+0000","last_unstale":"2024-03-26T21:38:01.175529+0000","last_undegraded":"2024-03-26T21:38:01.175529+0000","last_fullsized":"2024-03-26T21:38:01.175529+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'28","last_scrub_stamp":"2024-03-26T21:37:33.889037+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:37:33.889037+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:34.275460+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"227'16305","reported_seq":16715,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.962479+0000","last_change":"2024-03-26T21:37:18.304107+0000","last_active":"2024-03-26T21:38:01.962479+0000","last_peered":"2024-03-26T21:38:01.962479+0000","last_clean":"2024-03-26T21:38:01.962479+0000","last_became_active":"2024-03-26T21:34:58.693834+0000","last_became_peered":"2024-03-26T21:34:58.693834+0000","last_unstale":"2024-03-26T21:38:01.962479+0000","last_undegraded":"2024-03-26T21:38:01.962479+0000","last_fullsized":"2024-03-26T21:38:01.962479+0000","mapping_epoch":173,"log_start":"206'11360","ondisk_log_start":"206'11360","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13926","last_scrub_stamp":"2024-03-26T21:37:18.303864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:18.303864+0000","objects_scrubbed":5327,"log_size":4945,"log_dups_size":0,"ondisk_log_size":4945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:33.660416+0000","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416000,"num_objects":2875,"num_object_clones":0,"num_object_copies":8625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":459968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181333+0000","last_change":"2024-03-26T21:35:13.040636+0000","last_active":"2024-03-26T21:38:01.181333+0000","last_peered":"2024-03-26T21:38:01.181333+0000","last_clean":"2024-03-26T21:38:01.181333+0000","last_became_active":"2024-03-26T21:28:40.888886+0000","last_became_peered":"2024-03-26T21:28:40.888886+0000","last_unstale":"2024-03-26T21:38:01.181333+0000","last_undegraded":"2024-03-26T21:38:01.181333+0000","last_fullsized":"2024-03-26T21:38:01.181333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:13.040364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:35:13.040364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181222+0000","last_change":"2024-03-26T21:37:30.835062+0000","last_active":"2024-03-26T21:38:01.181222+0000","last_peered":"2024-03-26T21:38:01.181222+0000","last_clean":"2024-03-26T21:38:01.181222+0000","last_became_active":"2024-03-26T21:36:51.707172+0000","last_became_peered":"2024-03-26T21:36:51.707172+0000","last_unstale":"2024-03-26T21:38:01.181222+0000","last_undegraded":"2024-03-26T21:38:01.181222+0000","last_fullsized":"2024-03-26T21:38:01.181222+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:30.834604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:30.834604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:44.070832+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172906+0000","last_change":"2024-03-26T21:37:30.240886+0000","last_active":"2024-03-26T21:38:01.172906+0000","last_peered":"2024-03-26T21:38:01.172906+0000","last_clean":"2024-03-26T21:38:01.172906+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:38:01.172906+0000","last_undegraded":"2024-03-26T21:38:01.172906+0000","last_fullsized":"2024-03-26T21:38:01.172906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:30.240758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:37:30.240758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:37.163548+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":404,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181725+0000","last_change":"2024-03-26T21:37:32.461009+0000","last_active":"2024-03-26T21:38:01.181725+0000","last_peered":"2024-03-26T21:38:01.181725+0000","last_clean":"2024-03-26T21:38:01.181725+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:38:01.181725+0000","last_undegraded":"2024-03-26T21:38:01.181725+0000","last_fullsized":"2024-03-26T21:38:01.181725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-26T21:37:32.460870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:32.460870+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:55.410561+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"227'11393","reported_seq":11554,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.558502+0000","last_change":"2024-03-26T21:37:49.135325+0000","last_active":"2024-03-26T21:38:01.558502+0000","last_peered":"2024-03-26T21:38:01.558502+0000","last_clean":"2024-03-26T21:38:01.558502+0000","last_became_active":"2024-03-26T21:34:59.371434+0000","last_became_peered":"2024-03-26T21:34:59.371434+0000","last_unstale":"2024-03-26T21:38:01.558502+0000","last_undegraded":"2024-03-26T21:38:01.558502+0000","last_fullsized":"2024-03-26T21:38:01.558502+0000","mapping_epoch":173,"log_start":"192'6106","ondisk_log_start":"192'6106","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10890","last_scrub_stamp":"2024-03-26T21:37:49.135175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:49.135175+0000","objects_scrubbed":3360,"log_size":5287,"log_dups_size":0,"ondisk_log_size":5287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:16.829065+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185532416,"num_objects":2831,"num_object_clones":0,"num_object_copies":8493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":368,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.179029+0000","last_change":"2024-03-26T21:37:31.533161+0000","last_active":"2024-03-26T21:38:01.179029+0000","last_peered":"2024-03-26T21:38:01.179029+0000","last_clean":"2024-03-26T21:38:01.179029+0000","last_became_active":"2024-03-26T21:36:51.691238+0000","last_became_peered":"2024-03-26T21:36:51.691238+0000","last_unstale":"2024-03-26T21:38:01.179029+0000","last_undegraded":"2024-03-26T21:38:01.179029+0000","last_fullsized":"2024-03-26T21:38:01.179029+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:31.533053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:37:31.533053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:45.222269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":376,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.171183+0000","last_change":"2024-03-26T21:37:43.242135+0000","last_active":"2024-03-26T21:38:01.171183+0000","last_peered":"2024-03-26T21:38:01.171183+0000","last_clean":"2024-03-26T21:38:01.171183+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:38:01.171183+0000","last_undegraded":"2024-03-26T21:38:01.171183+0000","last_fullsized":"2024-03-26T21:38:01.171183+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-26T21:37:43.241985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:43.241985+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:50.418834+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":59,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141278+0000","last_change":"2024-03-26T21:37:41.893708+0000","last_active":"2024-03-26T21:37:58.141278+0000","last_peered":"2024-03-26T21:37:58.141278+0000","last_clean":"2024-03-26T21:37:58.141278+0000","last_became_active":"2024-03-26T21:36:51.695776+0000","last_became_peered":"2024-03-26T21:36:51.695776+0000","last_unstale":"2024-03-26T21:37:58.141278+0000","last_undegraded":"2024-03-26T21:37:58.141278+0000","last_fullsized":"2024-03-26T21:37:58.141278+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:41.893601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:41.893601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:47.535028+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"77'29","reported_seq":479,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172471+0000","last_change":"2024-03-26T21:37:29.264458+0000","last_active":"2024-03-26T21:38:01.172471+0000","last_peered":"2024-03-26T21:38:01.172471+0000","last_clean":"2024-03-26T21:38:01.172471+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:38:01.172471+0000","last_undegraded":"2024-03-26T21:38:01.172471+0000","last_fullsized":"2024-03-26T21:38:01.172471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'29","last_scrub_stamp":"2024-03-26T21:37:29.264322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:29.264322+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:52.393153+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181882+0000","last_change":"2024-03-26T21:37:34.439157+0000","last_active":"2024-03-26T21:38:01.181882+0000","last_peered":"2024-03-26T21:38:01.181882+0000","last_clean":"2024-03-26T21:38:01.181882+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:38:01.181882+0000","last_undegraded":"2024-03-26T21:38:01.181882+0000","last_fullsized":"2024-03-26T21:38:01.181882+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:34.439022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:34.439022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:36.960737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"227'11460","reported_seq":24304,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:02.263850+0000","last_change":"2024-03-26T21:37:28.831111+0000","last_active":"2024-03-26T21:38:02.263850+0000","last_peered":"2024-03-26T21:38:02.263850+0000","last_clean":"2024-03-26T21:38:02.263850+0000","last_became_active":"2024-03-26T21:37:28.759750+0000","last_became_peered":"2024-03-26T21:37:28.759750+0000","last_unstale":"2024-03-26T21:38:02.263850+0000","last_undegraded":"2024-03-26T21:38:02.263850+0000","last_fullsized":"2024-03-26T21:38:02.263850+0000","mapping_epoch":215,"log_start":"194'6522","ondisk_log_start":"194'6522","created":173,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6346,"num_bytes_recovered":415825920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182461+0000","last_change":"2024-03-26T21:35:27.571293+0000","last_active":"2024-03-26T21:38:01.182461+0000","last_peered":"2024-03-26T21:38:01.182461+0000","last_clean":"2024-03-26T21:38:01.182461+0000","last_became_active":"2024-03-26T21:28:41.113768+0000","last_became_peered":"2024-03-26T21:28:41.113768+0000","last_unstale":"2024-03-26T21:38:01.182461+0000","last_undegraded":"2024-03-26T21:38:01.182461+0000","last_fullsized":"2024-03-26T21:38:01.182461+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:27.571209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:35:27.571209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":383,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.176581+0000","last_change":"2024-03-26T21:37:39.503489+0000","last_active":"2024-03-26T21:38:01.176581+0000","last_peered":"2024-03-26T21:38:01.176581+0000","last_clean":"2024-03-26T21:38:01.176581+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:38:01.176581+0000","last_undegraded":"2024-03-26T21:38:01.176581+0000","last_fullsized":"2024-03-26T21:38:01.176581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:39.503106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:37:39.503106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:49.554194+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":374,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181145+0000","last_change":"2024-03-26T21:35:48.214395+0000","last_active":"2024-03-26T21:38:01.181145+0000","last_peered":"2024-03-26T21:38:01.181145+0000","last_clean":"2024-03-26T21:38:01.181145+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:38:01.181145+0000","last_undegraded":"2024-03-26T21:38:01.181145+0000","last_fullsized":"2024-03-26T21:38:01.181145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:48.214286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:35:48.214286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"88'18","reported_seq":403,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174680+0000","last_change":"2024-03-26T21:37:37.534467+0000","last_active":"2024-03-26T21:38:01.174680+0000","last_peered":"2024-03-26T21:38:01.174680+0000","last_clean":"2024-03-26T21:38:01.174680+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:38:01.174680+0000","last_undegraded":"2024-03-26T21:38:01.174680+0000","last_fullsized":"2024-03-26T21:38:01.174680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'18","last_scrub_stamp":"2024-03-26T21:37:37.534340+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:37:37.534340+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:40.017820+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182798+0000","last_change":"2024-03-26T21:36:51.137538+0000","last_active":"2024-03-26T21:38:01.182798+0000","last_peered":"2024-03-26T21:38:01.182798+0000","last_clean":"2024-03-26T21:38:01.182798+0000","last_became_active":"2024-03-26T21:28:38.873743+0000","last_became_peered":"2024-03-26T21:28:38.873743+0000","last_unstale":"2024-03-26T21:38:01.182798+0000","last_undegraded":"2024-03-26T21:38:01.182798+0000","last_fullsized":"2024-03-26T21:38:01.182798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:36:51.137327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:36:51.137327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:16.861332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"227'11562","reported_seq":11733,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.555645+0000","last_change":"2024-03-26T21:37:34.136602+0000","last_active":"2024-03-26T21:38:01.555645+0000","last_peered":"2024-03-26T21:38:01.555645+0000","last_clean":"2024-03-26T21:38:01.555645+0000","last_became_active":"2024-03-26T21:34:58.687512+0000","last_became_peered":"2024-03-26T21:34:58.687512+0000","last_unstale":"2024-03-26T21:38:01.555645+0000","last_undegraded":"2024-03-26T21:38:01.555645+0000","last_fullsized":"2024-03-26T21:38:01.555645+0000","mapping_epoch":173,"log_start":"191'5431","ondisk_log_start":"191'5431","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'10023","last_scrub_stamp":"2024-03-26T21:37:34.136253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:34.136253+0000","objects_scrubbed":4471,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:53.040041+0000","scrub_duration":1292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11562,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":368,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172677+0000","last_change":"2024-03-26T21:36:51.678430+0000","last_active":"2024-03-26T21:38:01.172677+0000","last_peered":"2024-03-26T21:38:01.172677+0000","last_clean":"2024-03-26T21:38:01.172677+0000","last_became_active":"2024-03-26T21:36:51.678088+0000","last_became_peered":"2024-03-26T21:36:51.678088+0000","last_unstale":"2024-03-26T21:38:01.172677+0000","last_undegraded":"2024-03-26T21:38:01.172677+0000","last_fullsized":"2024-03-26T21:38:01.172677+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:44.984553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:44.984553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188418+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":619,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181781+0000","last_change":"2024-03-26T21:37:24.027917+0000","last_active":"2024-03-26T21:38:01.181781+0000","last_peered":"2024-03-26T21:38:01.181781+0000","last_clean":"2024-03-26T21:38:01.181781+0000","last_became_active":"2024-03-26T21:28:24.728618+0000","last_became_peered":"2024-03-26T21:28:24.728618+0000","last_unstale":"2024-03-26T21:38:01.181781+0000","last_undegraded":"2024-03-26T21:38:01.181781+0000","last_fullsized":"2024-03-26T21:38:01.181781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:37:24.027768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:37:24.027768+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:44.691714+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.177663+0000","last_change":"2024-03-26T21:37:42.570702+0000","last_active":"2024-03-26T21:38:01.177663+0000","last_peered":"2024-03-26T21:38:01.177663+0000","last_clean":"2024-03-26T21:38:01.177663+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:38:01.177663+0000","last_undegraded":"2024-03-26T21:38:01.177663+0000","last_fullsized":"2024-03-26T21:38:01.177663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:42.570535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:42.570535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:00.648461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"205'28","reported_seq":567,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.176007+0000","last_change":"2024-03-26T21:37:27.838949+0000","last_active":"2024-03-26T21:38:01.176007+0000","last_peered":"2024-03-26T21:38:01.176007+0000","last_clean":"2024-03-26T21:38:01.176007+0000","last_became_active":"2024-03-26T21:36:51.696542+0000","last_became_peered":"2024-03-26T21:36:51.696542+0000","last_unstale":"2024-03-26T21:38:01.176007+0000","last_undegraded":"2024-03-26T21:38:01.176007+0000","last_fullsized":"2024-03-26T21:38:01.176007+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'28","last_scrub_stamp":"2024-03-26T21:37:27.838860+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:37:27.838860+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:53.618695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183457+0000","last_change":"2024-03-26T21:37:36.037227+0000","last_active":"2024-03-26T21:38:01.183457+0000","last_peered":"2024-03-26T21:38:01.183457+0000","last_clean":"2024-03-26T21:38:01.183457+0000","last_became_active":"2024-03-26T21:28:38.868382+0000","last_became_peered":"2024-03-26T21:28:38.868382+0000","last_unstale":"2024-03-26T21:38:01.183457+0000","last_undegraded":"2024-03-26T21:38:01.183457+0000","last_fullsized":"2024-03-26T21:38:01.183457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:36.036984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:36.036984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:49.168849+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"227'11349","reported_seq":11525,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.966890+0000","last_change":"2024-03-26T21:37:22.379937+0000","last_active":"2024-03-26T21:38:01.966890+0000","last_peered":"2024-03-26T21:38:01.966890+0000","last_clean":"2024-03-26T21:38:01.966890+0000","last_became_active":"2024-03-26T21:34:58.693101+0000","last_became_peered":"2024-03-26T21:34:58.693101+0000","last_unstale":"2024-03-26T21:38:01.966890+0000","last_undegraded":"2024-03-26T21:38:01.966890+0000","last_fullsized":"2024-03-26T21:38:01.966890+0000","mapping_epoch":173,"log_start":"193'6385","ondisk_log_start":"193'6385","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'9294","last_scrub_stamp":"2024-03-26T21:37:22.379733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:22.379733+0000","objects_scrubbed":4993,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:51.080287+0000","scrub_duration":1062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191823872,"num_objects":2927,"num_object_clones":0,"num_object_copies":8781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11349,"num_write_kb":456832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.173063+0000","last_change":"2024-03-26T21:37:31.484704+0000","last_active":"2024-03-26T21:38:01.173063+0000","last_peered":"2024-03-26T21:38:01.173063+0000","last_clean":"2024-03-26T21:38:01.173063+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:38:01.173063+0000","last_undegraded":"2024-03-26T21:38:01.173063+0000","last_fullsized":"2024-03-26T21:38:01.173063+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:31.484627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:31.484627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:46.703203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181122+0000","last_change":"2024-03-26T21:37:41.409039+0000","last_active":"2024-03-26T21:38:01.181122+0000","last_peered":"2024-03-26T21:38:01.181122+0000","last_clean":"2024-03-26T21:38:01.181122+0000","last_became_active":"2024-03-26T21:36:51.685071+0000","last_became_peered":"2024-03-26T21:36:51.685071+0000","last_unstale":"2024-03-26T21:38:01.181122+0000","last_undegraded":"2024-03-26T21:38:01.181122+0000","last_fullsized":"2024-03-26T21:38:01.181122+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:41.408914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:37:41.408914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:42.906373+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":945,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181733+0000","last_change":"2024-03-26T21:37:51.973275+0000","last_active":"2024-03-26T21:38:01.181733+0000","last_peered":"2024-03-26T21:38:01.181733+0000","last_clean":"2024-03-26T21:38:01.181733+0000","last_became_active":"2024-03-26T21:28:34.850285+0000","last_became_peered":"2024-03-26T21:28:34.850285+0000","last_unstale":"2024-03-26T21:38:01.181733+0000","last_undegraded":"2024-03-26T21:38:01.181733+0000","last_fullsized":"2024-03-26T21:38:01.181733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T21:37:51.973154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.973154+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:19.012395+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":560,"num_read_kb":555,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":401,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.187541+0000","last_change":"2024-03-26T21:37:30.529329+0000","last_active":"2024-03-26T21:38:01.187541+0000","last_peered":"2024-03-26T21:38:01.187541+0000","last_clean":"2024-03-26T21:38:01.187541+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:38:01.187541+0000","last_undegraded":"2024-03-26T21:38:01.187541+0000","last_fullsized":"2024-03-26T21:38:01.187541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:37:30.529179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:30.529179+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:55.555824+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172776+0000","last_change":"2024-03-26T21:37:34.223175+0000","last_active":"2024-03-26T21:38:01.172776+0000","last_peered":"2024-03-26T21:38:01.172776+0000","last_clean":"2024-03-26T21:38:01.172776+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:38:01.172776+0000","last_undegraded":"2024-03-26T21:38:01.172776+0000","last_fullsized":"2024-03-26T21:38:01.172776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:34.223039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:34.223039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:53.713559+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"227'11406","reported_seq":11525,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:02.282561+0000","last_change":"2024-03-26T21:34:58.693123+0000","last_active":"2024-03-26T21:38:02.282561+0000","last_peered":"2024-03-26T21:38:02.282561+0000","last_clean":"2024-03-26T21:38:02.282561+0000","last_became_active":"2024-03-26T21:34:58.692159+0000","last_became_peered":"2024-03-26T21:34:58.692159+0000","last_unstale":"2024-03-26T21:38:02.282561+0000","last_undegraded":"2024-03-26T21:38:02.282561+0000","last_fullsized":"2024-03-26T21:38:02.282561+0000","mapping_epoch":173,"log_start":"195'6532","ondisk_log_start":"195'6532","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183500800,"num_objects":2800,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11406,"num_write_kb":454592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174786+0000","last_change":"2024-03-26T21:37:29.531637+0000","last_active":"2024-03-26T21:38:01.174786+0000","last_peered":"2024-03-26T21:38:01.174786+0000","last_clean":"2024-03-26T21:38:01.174786+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:38:01.174786+0000","last_undegraded":"2024-03-26T21:38:01.174786+0000","last_fullsized":"2024-03-26T21:38:01.174786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:29.531508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:29.531508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:50.237066+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.173292+0000","last_change":"2024-03-26T21:37:30.479030+0000","last_active":"2024-03-26T21:38:01.173292+0000","last_peered":"2024-03-26T21:38:01.173292+0000","last_clean":"2024-03-26T21:38:01.173292+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:38:01.173292+0000","last_undegraded":"2024-03-26T21:38:01.173292+0000","last_fullsized":"2024-03-26T21:38:01.173292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:30.478388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:37:30.478388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:39.243936+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":391,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181594+0000","last_change":"2024-03-26T21:35:11.003728+0000","last_active":"2024-03-26T21:38:01.181594+0000","last_peered":"2024-03-26T21:38:01.181594+0000","last_clean":"2024-03-26T21:38:01.181594+0000","last_became_active":"2024-03-26T21:28:37.141402+0000","last_became_peered":"2024-03-26T21:28:37.141402+0000","last_unstale":"2024-03-26T21:38:01.181594+0000","last_undegraded":"2024-03-26T21:38:01.181594+0000","last_fullsized":"2024-03-26T21:38:01.181594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-26T21:35:11.003580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:11.003580+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.175441+0000","last_change":"2024-03-26T21:35:15.119981+0000","last_active":"2024-03-26T21:38:01.175441+0000","last_peered":"2024-03-26T21:38:01.175441+0000","last_clean":"2024-03-26T21:38:01.175441+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:38:01.175441+0000","last_undegraded":"2024-03-26T21:38:01.175441+0000","last_fullsized":"2024-03-26T21:38:01.175441+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.119862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.119862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"227'11405","reported_seq":18852,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:03.017812+0000","last_change":"2024-03-26T21:37:45.951923+0000","last_active":"2024-03-26T21:38:03.017812+0000","last_peered":"2024-03-26T21:38:03.017812+0000","last_clean":"2024-03-26T21:38:03.017812+0000","last_became_active":"2024-03-26T21:37:31.812842+0000","last_became_peered":"2024-03-26T21:37:31.812842+0000","last_unstale":"2024-03-26T21:38:03.017812+0000","last_undegraded":"2024-03-26T21:38:03.017812+0000","last_fullsized":"2024-03-26T21:38:03.017812+0000","mapping_epoch":217,"log_start":"192'5632","ondisk_log_start":"192'5632","created":173,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'10578","last_scrub_stamp":"2024-03-26T21:37:45.951565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:45.951565+0000","objects_scrubbed":3577,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.934739+0000","scrub_duration":1460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":8121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11405,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3634,"num_bytes_recovered":211681280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.180403+0000","last_change":"2024-03-26T21:35:30.917840+0000","last_active":"2024-03-26T21:38:01.180403+0000","last_peered":"2024-03-26T21:38:01.180403+0000","last_clean":"2024-03-26T21:38:01.180403+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:38:01.180403+0000","last_undegraded":"2024-03-26T21:38:01.180403+0000","last_fullsized":"2024-03-26T21:38:01.180403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:30.917720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:30.917720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":367,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172614+0000","last_change":"2024-03-26T21:35:25.841787+0000","last_active":"2024-03-26T21:38:01.172614+0000","last_peered":"2024-03-26T21:38:01.172614+0000","last_clean":"2024-03-26T21:38:01.172614+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:38:01.172614+0000","last_undegraded":"2024-03-26T21:38:01.172614+0000","last_fullsized":"2024-03-26T21:38:01.172614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:25.841708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:35:25.841708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:04.188286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":423,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178485+0000","last_change":"2024-03-26T21:37:28.479656+0000","last_active":"2024-03-26T21:38:01.178485+0000","last_peered":"2024-03-26T21:38:01.178485+0000","last_clean":"2024-03-26T21:38:01.178485+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:38:01.178485+0000","last_undegraded":"2024-03-26T21:38:01.178485+0000","last_fullsized":"2024-03-26T21:38:01.178485+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-26T21:37:28.479571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:28.479571+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:47.732582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":473,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.181371+0000","last_change":"2024-03-26T21:35:28.921519+0000","last_active":"2024-03-26T21:38:01.181371+0000","last_peered":"2024-03-26T21:38:01.181371+0000","last_clean":"2024-03-26T21:38:01.181371+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:38:01.181371+0000","last_undegraded":"2024-03-26T21:38:01.181371+0000","last_fullsized":"2024-03-26T21:38:01.181371+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:35:28.921392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.921392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"227'11262","reported_seq":11423,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.510288+0000","last_change":"2024-03-26T21:37:50.910360+0000","last_active":"2024-03-26T21:38:01.510288+0000","last_peered":"2024-03-26T21:38:01.510288+0000","last_clean":"2024-03-26T21:38:01.510288+0000","last_became_active":"2024-03-26T21:34:59.371634+0000","last_became_peered":"2024-03-26T21:34:59.371634+0000","last_unstale":"2024-03-26T21:38:01.510288+0000","last_undegraded":"2024-03-26T21:38:01.510288+0000","last_fullsized":"2024-03-26T21:38:01.510288+0000","mapping_epoch":173,"log_start":"192'5900","ondisk_log_start":"192'5900","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'10872","last_scrub_stamp":"2024-03-26T21:37:50.910014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:50.910014+0000","objects_scrubbed":3310,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:11.815887+0000","scrub_duration":661,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190382104,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11262,"num_write_kb":453313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182433+0000","last_change":"2024-03-26T21:35:10.438550+0000","last_active":"2024-03-26T21:38:01.182433+0000","last_peered":"2024-03-26T21:38:01.182433+0000","last_clean":"2024-03-26T21:38:01.182433+0000","last_became_active":"2024-03-26T21:28:41.115531+0000","last_became_peered":"2024-03-26T21:28:41.115531+0000","last_unstale":"2024-03-26T21:38:01.182433+0000","last_undegraded":"2024-03-26T21:38:01.182433+0000","last_fullsized":"2024-03-26T21:38:01.182433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:10.438407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:10.438407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":381,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.187649+0000","last_change":"2024-03-26T21:37:31.476545+0000","last_active":"2024-03-26T21:38:01.187649+0000","last_peered":"2024-03-26T21:38:01.187649+0000","last_clean":"2024-03-26T21:38:01.187649+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:38:01.187649+0000","last_undegraded":"2024-03-26T21:38:01.187649+0000","last_fullsized":"2024-03-26T21:38:01.187649+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:37:31.476459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:31.476459+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:54.967097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":67,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.139696+0000","last_change":"2024-03-26T21:36:51.743555+0000","last_active":"2024-03-26T21:37:58.139696+0000","last_peered":"2024-03-26T21:37:58.139696+0000","last_clean":"2024-03-26T21:37:58.139696+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:37:58.139696+0000","last_undegraded":"2024-03-26T21:37:58.139696+0000","last_fullsized":"2024-03-26T21:37:58.139696+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-26T21:35:26.141418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:26.141418+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.742958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"205'12","reported_seq":505,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178373+0000","last_change":"2024-03-26T21:36:51.724295+0000","last_active":"2024-03-26T21:38:01.178373+0000","last_peered":"2024-03-26T21:38:01.178373+0000","last_clean":"2024-03-26T21:38:01.178373+0000","last_became_active":"2024-03-26T21:36:51.690572+0000","last_became_peered":"2024-03-26T21:36:51.690572+0000","last_unstale":"2024-03-26T21:38:01.178373+0000","last_undegraded":"2024-03-26T21:38:01.178373+0000","last_fullsized":"2024-03-26T21:38:01.178373+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'11","last_scrub_stamp":"2024-03-26T21:36:49.458409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:36:49.458409+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:05.721361+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"227'13902","reported_seq":14117,"reported_epoch":227,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-26T21:38:03.045628+0000","last_change":"2024-03-26T21:37:49.643038+0000","last_active":"2024-03-26T21:38:03.045628+0000","last_peered":"2024-03-26T21:38:03.045628+0000","last_clean":"2024-03-26T21:38:03.045628+0000","last_became_active":"2024-03-26T21:34:59.370503+0000","last_became_peered":"2024-03-26T21:34:59.370503+0000","last_unstale":"2024-03-26T21:38:03.045628+0000","last_undegraded":"2024-03-26T21:38:03.045628+0000","last_fullsized":"2024-03-26T21:38:03.045628+0000","mapping_epoch":173,"log_start":"207'8056","ondisk_log_start":"207'8056","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":2423,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187629568,"num_objects":2863,"num_object_clones":0,"num_object_copies":8589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11479,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":59,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.140403+0000","last_change":"2024-03-26T21:37:34.876679+0000","last_active":"2024-03-26T21:37:58.140403+0000","last_peered":"2024-03-26T21:37:58.140403+0000","last_clean":"2024-03-26T21:37:58.140403+0000","last_became_active":"2024-03-26T21:36:51.696247+0000","last_became_peered":"2024-03-26T21:36:51.696247+0000","last_unstale":"2024-03-26T21:37:58.140403+0000","last_undegraded":"2024-03-26T21:37:58.140403+0000","last_fullsized":"2024-03-26T21:37:58.140403+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:34.876568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:37:34.876568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:51.819548+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.183278+0000","last_change":"2024-03-26T21:35:28.245063+0000","last_active":"2024-03-26T21:38:01.183278+0000","last_peered":"2024-03-26T21:38:01.183278+0000","last_clean":"2024-03-26T21:38:01.183278+0000","last_became_active":"2024-03-26T21:28:34.839525+0000","last_became_peered":"2024-03-26T21:28:34.839525+0000","last_unstale":"2024-03-26T21:38:01.183278+0000","last_undegraded":"2024-03-26T21:38:01.183278+0000","last_fullsized":"2024-03-26T21:38:01.183278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:28.244684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:28.244684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735079+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":387,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178271+0000","last_change":"2024-03-26T21:35:33.264772+0000","last_active":"2024-03-26T21:38:01.178271+0000","last_peered":"2024-03-26T21:38:01.178271+0000","last_clean":"2024-03-26T21:38:01.178271+0000","last_became_active":"2024-03-26T21:28:37.142441+0000","last_became_peered":"2024-03-26T21:28:37.142441+0000","last_unstale":"2024-03-26T21:38:01.178271+0000","last_undegraded":"2024-03-26T21:38:01.178271+0000","last_fullsized":"2024-03-26T21:38:01.178271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:35:33.264627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.264627+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.735263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.139886+0000","last_change":"2024-03-26T21:36:51.682642+0000","last_active":"2024-03-26T21:37:58.139886+0000","last_peered":"2024-03-26T21:37:58.139886+0000","last_clean":"2024-03-26T21:37:58.139886+0000","last_became_active":"2024-03-26T21:36:51.682044+0000","last_became_peered":"2024-03-26T21:36:51.682044+0000","last_unstale":"2024-03-26T21:37:58.139886+0000","last_undegraded":"2024-03-26T21:37:58.139886+0000","last_fullsized":"2024-03-26T21:37:58.139886+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:44.244504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:44.244504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"227'11397","reported_seq":11577,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.945194+0000","last_change":"2024-03-26T21:37:20.640140+0000","last_active":"2024-03-26T21:38:01.945194+0000","last_peered":"2024-03-26T21:38:01.945194+0000","last_clean":"2024-03-26T21:38:01.945194+0000","last_became_active":"2024-03-26T21:34:58.690860+0000","last_became_peered":"2024-03-26T21:34:58.690860+0000","last_unstale":"2024-03-26T21:38:01.945194+0000","last_undegraded":"2024-03-26T21:38:01.945194+0000","last_fullsized":"2024-03-26T21:38:01.945194+0000","mapping_epoch":173,"log_start":"193'6395","ondisk_log_start":"193'6395","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'9145","last_scrub_stamp":"2024-03-26T21:37:20.639981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:37:20.639981+0000","objects_scrubbed":5184,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:22.772327+0000","scrub_duration":2305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187891712,"num_objects":2867,"num_object_clones":0,"num_object_copies":8601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":456448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172344+0000","last_change":"2024-03-26T21:37:32.307860+0000","last_active":"2024-03-26T21:38:01.172344+0000","last_peered":"2024-03-26T21:38:01.172344+0000","last_clean":"2024-03-26T21:38:01.172344+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:38:01.172344+0000","last_undegraded":"2024-03-26T21:38:01.172344+0000","last_fullsized":"2024-03-26T21:38:01.172344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:32.307748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:37:32.307748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:56.243157+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":77,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141772+0000","last_change":"2024-03-26T21:37:46.912860+0000","last_active":"2024-03-26T21:37:58.141772+0000","last_peered":"2024-03-26T21:37:58.141772+0000","last_clean":"2024-03-26T21:37:58.141772+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:37:58.141772+0000","last_undegraded":"2024-03-26T21:37:58.141772+0000","last_fullsized":"2024-03-26T21:37:58.141772+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:37:46.912710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:46.912710+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:48.076448+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178169+0000","last_change":"2024-03-26T21:35:14.322330+0000","last_active":"2024-03-26T21:38:01.178169+0000","last_peered":"2024-03-26T21:38:01.178169+0000","last_clean":"2024-03-26T21:38:01.178169+0000","last_became_active":"2024-03-26T21:28:34.847419+0000","last_became_peered":"2024-03-26T21:28:34.847419+0000","last_unstale":"2024-03-26T21:38:01.178169+0000","last_undegraded":"2024-03-26T21:38:01.178169+0000","last_fullsized":"2024-03-26T21:38:01.178169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:14.322126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:35:14.322126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:01.734966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":441,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.174512+0000","last_change":"2024-03-26T21:36:51.787462+0000","last_active":"2024-03-26T21:38:01.174512+0000","last_peered":"2024-03-26T21:38:01.174512+0000","last_clean":"2024-03-26T21:38:01.174512+0000","last_became_active":"2024-03-26T21:36:51.716671+0000","last_became_peered":"2024-03-26T21:36:51.716671+0000","last_unstale":"2024-03-26T21:38:01.174512+0000","last_undegraded":"2024-03-26T21:38:01.174512+0000","last_fullsized":"2024-03-26T21:38:01.174512+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-26T21:36:42.254966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:36:42.254966+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"224'11312","reported_seq":24045,"reported_epoch":225,"state":"peering","last_fresh":"2024-03-26T21:37:58.141021+0000","last_change":"2024-03-26T21:37:58.141021+0000","last_active":"2024-03-26T21:37:58.141021+0000","last_peered":"2024-03-26T21:37:58.114725+0000","last_clean":"2024-03-26T21:37:58.114725+0000","last_became_active":"2024-03-26T21:36:52.744413+0000","last_became_peered":"2024-03-26T21:36:52.744413+0000","last_unstale":"2024-03-26T21:37:58.141021+0000","last_undegraded":"2024-03-26T21:37:58.141021+0000","last_fullsized":"2024-03-26T21:37:58.141021+0000","mapping_epoch":225,"log_start":"192'5538","ondisk_log_start":"192'5538","created":173,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6295,"num_bytes_recovered":412549120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182717+0000","last_change":"2024-03-26T21:37:52.675768+0000","last_active":"2024-03-26T21:38:01.182717+0000","last_peered":"2024-03-26T21:38:01.182717+0000","last_clean":"2024-03-26T21:38:01.182717+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:38:01.182717+0000","last_undegraded":"2024-03-26T21:38:01.182717+0000","last_fullsized":"2024-03-26T21:38:01.182717+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:52.675543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:37:52.675543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.518095+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.172532+0000","last_change":"2024-03-26T21:37:52.281963+0000","last_active":"2024-03-26T21:38:01.172532+0000","last_peered":"2024-03-26T21:38:01.172532+0000","last_clean":"2024-03-26T21:38:01.172532+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:38:01.172532+0000","last_undegraded":"2024-03-26T21:38:01.172532+0000","last_fullsized":"2024-03-26T21:38:01.172532+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:52.281838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:52.281838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:56.186270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.139604+0000","last_change":"2024-03-26T21:36:51.692684+0000","last_active":"2024-03-26T21:37:58.139604+0000","last_peered":"2024-03-26T21:37:58.139604+0000","last_clean":"2024-03-26T21:37:58.139604+0000","last_became_active":"2024-03-26T21:36:51.692267+0000","last_became_peered":"2024-03-26T21:36:51.692267+0000","last_unstale":"2024-03-26T21:37:58.139604+0000","last_undegraded":"2024-03-26T21:37:58.139604+0000","last_fullsized":"2024-03-26T21:37:58.139604+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:47.273182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:35:47.273182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":71,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.142054+0000","last_change":"2024-03-26T21:37:49.905844+0000","last_active":"2024-03-26T21:37:58.142054+0000","last_peered":"2024-03-26T21:37:58.142054+0000","last_clean":"2024-03-26T21:37:58.142054+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:37:58.142054+0000","last_undegraded":"2024-03-26T21:37:58.142054+0000","last_fullsized":"2024-03-26T21:37:58.142054+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-26T21:37:49.905749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:49.905749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:12.103347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"227'11520","reported_seq":11716,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.966596+0000","last_change":"2024-03-26T21:36:46.675407+0000","last_active":"2024-03-26T21:38:01.966596+0000","last_peered":"2024-03-26T21:38:01.966596+0000","last_clean":"2024-03-26T21:38:01.966596+0000","last_became_active":"2024-03-26T21:34:58.693174+0000","last_became_peered":"2024-03-26T21:34:58.693174+0000","last_unstale":"2024-03-26T21:38:01.966596+0000","last_undegraded":"2024-03-26T21:38:01.966596+0000","last_fullsized":"2024-03-26T21:38:01.966596+0000","mapping_epoch":173,"log_start":"195'6560","ondisk_log_start":"195'6560","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'7456","last_scrub_stamp":"2024-03-26T21:36:46.674962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:36:46.674962+0000","objects_scrubbed":6963,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:10.313255+0000","scrub_duration":2526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186777600,"num_objects":2850,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":459840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":376,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182397+0000","last_change":"2024-03-26T21:37:43.774366+0000","last_active":"2024-03-26T21:38:01.182397+0000","last_peered":"2024-03-26T21:38:01.182397+0000","last_clean":"2024-03-26T21:38:01.182397+0000","last_became_active":"2024-03-26T21:36:51.698823+0000","last_became_peered":"2024-03-26T21:36:51.698823+0000","last_unstale":"2024-03-26T21:38:01.182397+0000","last_undegraded":"2024-03-26T21:38:01.182397+0000","last_fullsized":"2024-03-26T21:38:01.182397+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:43.773919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:37:43.773919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:58.824484+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":388,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.173427+0000","last_change":"2024-03-26T21:37:34.503278+0000","last_active":"2024-03-26T21:38:01.173427+0000","last_peered":"2024-03-26T21:38:01.173427+0000","last_clean":"2024-03-26T21:38:01.173427+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:38:01.173427+0000","last_undegraded":"2024-03-26T21:38:01.173427+0000","last_fullsized":"2024-03-26T21:38:01.173427+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:34.503164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:34.503164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:40.480473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":388,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.175239+0000","last_change":"2024-03-26T21:37:28.822291+0000","last_active":"2024-03-26T21:38:01.175239+0000","last_peered":"2024-03-26T21:38:01.175239+0000","last_clean":"2024-03-26T21:38:01.175239+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:38:01.175239+0000","last_undegraded":"2024-03-26T21:38:01.175239+0000","last_fullsized":"2024-03-26T21:38:01.175239+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:28.822182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:37:28.822182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:38.427601+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":77,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141863+0000","last_change":"2024-03-26T21:37:40.879220+0000","last_active":"2024-03-26T21:37:58.141863+0000","last_peered":"2024-03-26T21:37:58.141863+0000","last_clean":"2024-03-26T21:37:58.141863+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:37:58.141863+0000","last_undegraded":"2024-03-26T21:37:58.141863+0000","last_fullsized":"2024-03-26T21:37:58.141863+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-26T21:37:40.879100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:40.879100+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:39:03.983493+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"227'11240","reported_seq":11359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:02.302857+0000","last_change":"2024-03-26T21:34:58.692961+0000","last_active":"2024-03-26T21:38:02.302857+0000","last_peered":"2024-03-26T21:38:02.302857+0000","last_clean":"2024-03-26T21:38:02.302857+0000","last_became_active":"2024-03-26T21:34:58.692182+0000","last_became_peered":"2024-03-26T21:34:58.692182+0000","last_unstale":"2024-03-26T21:38:02.302857+0000","last_undegraded":"2024-03-26T21:38:02.302857+0000","last_fullsized":"2024-03-26T21:38:02.302857+0000","mapping_epoch":173,"log_start":"193'6323","ondisk_log_start":"193'6323","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:57.654033+0000","last_clean_scrub_stamp":"2024-03-26T21:34:57.654033+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695104,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11240,"num_write_kb":451328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141639+0000","last_change":"2024-03-26T21:36:51.693272+0000","last_active":"2024-03-26T21:37:58.141639+0000","last_peered":"2024-03-26T21:37:58.141639+0000","last_clean":"2024-03-26T21:37:58.141639+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:37:58.141639+0000","last_undegraded":"2024-03-26T21:37:58.141639+0000","last_fullsized":"2024-03-26T21:37:58.141639+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:15.211195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:15.211195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.742977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":489,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.182162+0000","last_change":"2024-03-26T21:37:52.659941+0000","last_active":"2024-03-26T21:38:01.182162+0000","last_peered":"2024-03-26T21:38:01.182162+0000","last_clean":"2024-03-26T21:38:01.182162+0000","last_became_active":"2024-03-26T21:28:38.866472+0000","last_became_peered":"2024-03-26T21:28:38.866472+0000","last_unstale":"2024-03-26T21:38:01.182162+0000","last_undegraded":"2024-03-26T21:38:01.182162+0000","last_fullsized":"2024-03-26T21:38:01.182162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:37:52.659812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:37:52.659812+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:57.702942+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-26T21:38:01.178348+0000","last_change":"2024-03-26T21:37:35.574887+0000","last_active":"2024-03-26T21:38:01.178348+0000","last_peered":"2024-03-26T21:38:01.178348+0000","last_clean":"2024-03-26T21:38:01.178348+0000","last_became_active":"2024-03-26T21:36:51.691808+0000","last_became_peered":"2024-03-26T21:36:51.691808+0000","last_unstale":"2024-03-26T21:38:01.178348+0000","last_undegraded":"2024-03-26T21:38:01.178348+0000","last_fullsized":"2024-03-26T21:38:01.178348+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:35.574719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:35.574719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:59.181976+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"87'15","reported_seq":63,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-26T21:37:58.141635+0000","last_change":"2024-03-26T21:36:51.833720+0000","last_active":"2024-03-26T21:37:58.141635+0000","last_peered":"2024-03-26T21:37:58.141635+0000","last_clean":"2024-03-26T21:37:58.141635+0000","last_became_active":"2024-03-26T21:36:51.689830+0000","last_became_peered":"2024-03-26T21:36:51.689830+0000","last_unstale":"2024-03-26T21:37:58.141635+0000","last_undegraded":"2024-03-26T21:37:58.141635+0000","last_fullsized":"2024-03-26T21:37:58.141635+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'15","last_scrub_stamp":"2024-03-26T21:35:43.214737+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:35:43.214737+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:38:02.743117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2998599704,"num_objects":45756,"num_object_clones":0,"num_object_copies":137268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182178,"num_write_kb":7293825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25590,"num_bytes_recovered":1593966592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9376378880,"data_stored":9376366664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86131,"ondisk_log_size":86131,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":560,"num_read_kb":555,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070582,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266588,"kb_used_data":1050768,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92478564,"statfs":{"total":95995035648,"available":94698049536,"internally_reserved":0,"allocated":1075986432,"data_stored":1075563145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364924,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1405684,"kb_used_data":1139884,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92339468,"statfs":{"total":95995035648,"available":94555615232,"internally_reserved":0,"allocated":1167241216,"data_stored":1166852357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1588596,"kb_used_data":1300520,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92156556,"statfs":{"total":95995035648,"available":94368313344,"internally_reserved":0,"allocated":1331732480,"data_stored":1331290374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738523,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1424800,"kb_used_data":1078488,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92320352,"statfs":{"total":95995035648,"available":94536040448,"internally_reserved":0,"allocated":1104371712,"data_stored":1103946389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411472,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399552,"kb_used_data":1101812,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92345600,"statfs":{"total":95995035648,"available":94561894400,"internally_reserved":0,"allocated":1128255488,"data_stored":1127864906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507624,"kb_used_data":1107164,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92237528,"statfs":{"total":95995035648,"available":94451228672,"internally_reserved":0,"allocated":1133735936,"data_stored":1133305159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":410057159},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650430,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1224896,"kb_used_data":967860,"kb_used_omap":4,"kb_used_meta":257019,"kb_avail":92520256,"statfs":{"total":95995035648,"available":94740742144,"internally_reserved":0,"allocated":991088640,"data_stored":990679312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":263188131},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659263,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1718940,"kb_used_data":1489228,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92026212,"statfs":{"total":95995035648,"available":94234841088,"internally_reserved":0,"allocated":1524969472,"data_stored":1524554883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1094713344,"data_stored":1094713344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1318715392,"data_stored":1318715392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1125191680,"data_stored":1125187608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1123155968,"data_stored":1123155968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":980942848,"data_stored":980942848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1512968192,"data_stored":1512964120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1153499136,"data_stored":1153499136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1067192320,"data_stored":1067188248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:38:04.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-03-26T21:38:05.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:05.064 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:05.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:05.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:05.324 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:05.413 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:05.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:05.432 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:05 smithi055 ceph-mon[29406]: pgmap v412: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 981 KiB/s, 15 objects/s recovering 2024-03-26T21:38:05.432 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:05 smithi055 ceph-mon[29406]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:38:05.432 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:05 smithi055 ceph-mon[40587]: pgmap v412: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 981 KiB/s, 15 objects/s recovering 2024-03-26T21:38:05.432 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:05 smithi055 ceph-mon[40587]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:38:05.451 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:05.487 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:05 smithi067 ceph-mon[33346]: pgmap v412: 148 pgs: 1 peering, 2 active+clean+scrubbing+deep, 145 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 639 op/s; 981 KiB/s, 15 objects/s recovering 2024-03-26T21:38:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:05 smithi067 ceph-mon[33346]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:38:05.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:05.591 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:05.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.727 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:05.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:05.828 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:05.863 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:05.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:05.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:06.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:06.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:06.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:06.239 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:06 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:38:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:06 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:38:06.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:06 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:38:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:06 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:38:06.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:06.396 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:06 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:38:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:06 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:38:06.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:06.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:06.678 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:06.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:06.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:06.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:06.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:07.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:07.088 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:07.190 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:07.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:07 smithi067 ceph-mon[33346]: pgmap v413: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 0 B/s wr, 465 op/s 2024-03-26T21:38:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:07 smithi067 ceph-mon[33346]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:38:07.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:07.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:07.370 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:07.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:07 smithi055 ceph-mon[29406]: pgmap v413: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 0 B/s wr, 465 op/s 2024-03-26T21:38:07.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:07 smithi055 ceph-mon[29406]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:38:07.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:07 smithi055 ceph-mon[40587]: pgmap v413: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 698 B/s rd, 0 B/s wr, 465 op/s 2024-03-26T21:38:07.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:07 smithi055 ceph-mon[40587]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T21:38:07.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:38:07.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:07.522 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:07.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:07.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:07.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:07.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:07.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:07.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:08.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:08.067 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:08.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:08.208 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:08.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:08.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:08.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:08.483 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:08 smithi067 ceph-mon[33346]: 8.c deep-scrub ok 2024-03-26T21:38:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:08 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:38:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:08 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:38:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:08 smithi067 ceph-mon[33346]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:38:08.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:08.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[40587]: 8.c deep-scrub ok 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[40587]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[29406]: 8.c deep-scrub ok 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:38:08.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-mon[29406]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T21:38:08.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:08.641 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:08.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:08.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:08.876 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:38:08.876 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:38:08.876 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:38:08.876 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:38:08.876 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:38:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:08.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:09.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:09.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:09.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:09.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:09.175 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:09.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:09.314 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:09.343 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:09 smithi055 ceph-mon[40587]: pgmap v415: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s 2024-03-26T21:38:09.343 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:09 smithi055 ceph-mon[40587]: 8.6 deep-scrub ok 2024-03-26T21:38:09.343 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:09 smithi055 ceph-mon[29406]: pgmap v415: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s 2024-03-26T21:38:09.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:09 smithi055 ceph-mon[29406]: 8.6 deep-scrub ok 2024-03-26T21:38:09.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:09 smithi067 ceph-mon[33346]: pgmap v415: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s 2024-03-26T21:38:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:09 smithi067 ceph-mon[33346]: 8.6 deep-scrub ok 2024-03-26T21:38:09.550 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:09.594 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:09.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:09.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:09.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:09.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:09.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:09.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:10.009 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:10.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:10.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:10.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/485028345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:10 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.269 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/485028345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.269 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:10 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.269 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 8 pgp_num to 16 2024-03-26T21:38:10.293 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:10.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/485028345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:10 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T21:38:10.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:10.561 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:10.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:10.694 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:10.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.831 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:10.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:10.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:10.966 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:10.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:11.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:11.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[40587]: pgmap v417: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-03-26T21:38:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T21:38:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[40587]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:38:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[29406]: pgmap v417: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-03-26T21:38:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T21:38:11.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:11 smithi055 ceph-mon[29406]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:38:11.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:11.404 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:11 smithi067 ceph-mon[33346]: pgmap v417: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-03-26T21:38:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T21:38:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:11 smithi067 ceph-mon[33346]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T21:38:11.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:11.549 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:11.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:11.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:11.791 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:11.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:11.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:11.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:11.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:12.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:12.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:12.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:38:12.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.260 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:12.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[29406]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[29406]: pgmap v420: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 784 op/s 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:38:12.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:38:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[40587]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:38:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:12 smithi055 ceph-mon[40587]: pgmap v420: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 784 op/s 2024-03-26T21:38:12.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:12.394 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:12.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:12.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:12.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:12.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:12 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:38:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:12 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:38:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:12 smithi067 ceph-mon[33346]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T21:38:12.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:12 smithi067 ceph-mon[33346]: pgmap v420: 148 pgs: 148 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 784 op/s 2024-03-26T21:38:12.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:12.785 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:12.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:12.926 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:12.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:13.021 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:13.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:13.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:13.197 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.245 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:13.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:13.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:13.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:13.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:13.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:13.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:13.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:13.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:13.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:13.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:13.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:14.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:14.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:14.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:14.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:14.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:14.427 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:14.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:14.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:14.537 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:38:14.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:14.633 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:14.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:14.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:14.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:14 smithi055 ceph-mon[29406]: pgmap v421: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T21:38:14.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:14.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:14 smithi055 ceph-mon[40587]: pgmap v421: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T21:38:14.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:14.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:14.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:14 smithi067 ceph-mon[33346]: pgmap v421: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-26T21:38:14.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:15.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:15.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:15.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:15.187 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:15.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:15.320 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:15.422 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:15.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:15.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:15.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:15.722 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:15.777 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:15.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:15.778 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[40587]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:38:15.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:15.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:15.778 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:15 smithi055 ceph-mon[29406]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:38:15.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:15.869 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:15.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:15.970 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:15 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:15 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2972814971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:15.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:15 smithi067 ceph-mon[33346]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T21:38:16.016 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:16.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:16.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:16.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:16.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:16.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:16.436 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:16.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:16.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.699 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:16.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:16.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:16.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:16 smithi055 ceph-mon[40587]: pgmap v423: 148 pgs: 148 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s 2024-03-26T21:38:16.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:16 smithi055 ceph-mon[29406]: pgmap v423: 148 pgs: 148 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s 2024-03-26T21:38:16.836 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:16.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:16.955 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:16 smithi067 ceph-mon[33346]: pgmap v423: 148 pgs: 148 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 932 op/s 2024-03-26T21:38:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:16.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:17.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:17.104 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:17.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:17.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:17.242 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:17.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.376 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:17.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:17.477 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:17.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:17.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:17.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:17.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:17.753 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:17.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:17.753 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:17.766 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:38:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:17.807 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:38:17.822 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:38:17.846 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:38:17.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:17.912 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:17.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:17.916 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:38:17.959 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:38:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:17.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:18.007 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:38:18.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:18.057 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:38:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.059 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:18.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:18.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:38:18.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:38:18.064 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-26T21:38:18.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-26T21:38:18.105 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:38:18.116 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:38:18.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:18.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:18.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:18.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:18.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:18.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:18.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:18.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:18.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:18.879 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:18.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:18.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:18 smithi055 ceph-mon[40587]: pgmap v424: 148 pgs: 148 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/628550639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:18.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:18 smithi055 ceph-mon[29406]: pgmap v424: 148 pgs: 148 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:18.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/628550639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:18 smithi067 ceph-mon[33346]: pgmap v424: 148 pgs: 148 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:18.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/628550639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:19.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:19.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:19.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:19.267 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:19.293 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:19.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:19.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:19.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:19.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:19.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:19.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:19.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:20.053 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:20.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:20.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:20.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:20.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:20.361 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:20.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:20.495 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:20.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:20.685 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:20 smithi055 ceph-mon[29406]: pgmap v425: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:20.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:20.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:20.899 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:20.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:20.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:20 smithi067 ceph-mon[33346]: pgmap v425: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:21.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.040 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:21.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:20 smithi055 ceph-mon[40587]: pgmap v425: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-26T21:38:21.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:21.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:21.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:21.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:21.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:21.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:21.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:21.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:21.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:21.839 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:21.843 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:21.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:21.936 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:21.936 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:21.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:21.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:22.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:22.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:22.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:22.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:22.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:22.491 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:22.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:22.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:22.770 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:22.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:22.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:22 smithi067 ceph-mon[33346]: pgmap v426: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T21:38:23.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:23.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:22 smithi055 ceph-mon[40587]: pgmap v426: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T21:38:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:22 smithi055 ceph-mon[29406]: pgmap v426: 148 pgs: 148 active+clean; 1.8 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T21:38:23.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:23.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:38:23.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:23.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:38:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:23.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:23.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.572 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:23.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:23.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:23.702 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:23.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:23.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:23.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:23.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:23.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:24.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:24.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:24.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:24.262 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:24.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:24.395 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:24.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:24.528 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:24.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.673 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:24.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:24.774 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:24.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:24 smithi055 ceph-mon[29406]: pgmap v427: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:38:24.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:24 smithi055 ceph-mon[40587]: pgmap v427: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:38:24.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:24.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:24.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:24.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:24.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:24 smithi067 ceph-mon[33346]: pgmap v427: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:38:25.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:25.108 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:25.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:25.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:25.373 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:25.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:25.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:25.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:25.777 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:25.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:25.916 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:25.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:26.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:26.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:26.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:26.195 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:26.207 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:26.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:26.329 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:26.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:26.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:26.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:26.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:26.753 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:26.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:26.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:26.969 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:26 smithi055 ceph-mon[40587]: pgmap v428: 148 pgs: 148 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T21:38:26.972 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:26 smithi055 ceph-mon[29406]: pgmap v428: 148 pgs: 148 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T21:38:26.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:26 smithi067 ceph-mon[33346]: pgmap v428: 148 pgs: 148 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T21:38:26.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:26 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:27.038 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:27.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:27.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:27.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:27.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:27.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:27.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:27.569 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:27.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:27.703 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:27.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.847 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:27.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:27.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:27.985 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:27.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:27 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:28.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:27 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:27 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:28.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:28.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:28.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:28.252 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:28.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:28.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:28.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:28.755 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.801 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:28.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:28.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:28 smithi055 ceph-mon[40587]: pgmap v429: 148 pgs: 148 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T21:38:28.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:28 smithi055 ceph-mon[29406]: pgmap v429: 148 pgs: 148 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T21:38:28.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:28.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:28.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:28 smithi067 ceph-mon[33346]: pgmap v429: 148 pgs: 148 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T21:38:29.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:29.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:29.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:29.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:29.368 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:29.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:29.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:29.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:29.773 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:38:29.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:29.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:29.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:29.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:29.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/722988008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:29.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:29 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:29.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/722988008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:29.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:29 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:30.035 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:30.073 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:30.079 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:30.080 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:30.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/722988008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:29 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:38:30.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:30.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:30.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:30.478 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:30.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:30.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:30.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:30.758 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:30.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:30.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:30.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:31.030 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: pgmap v430: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-26T21:38:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:31.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: pgmap v430: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:38:31.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:38:31.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:38:31.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:30 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:38:31.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: pgmap v430: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:38:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:30 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:38:31.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:31.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:31.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:31.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:31.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:31.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:31.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:31.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:31.796 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:31.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:31.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-26T21:38:31.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:31.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:31.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:31.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:31.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:32.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:31 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:32.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:31 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:32.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:32.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:31 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:32.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:32.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:32.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:32.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:32.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.558 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:32.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:32.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:32.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:32.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:32.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:32.966 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:32.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[40587]: pgmap v432: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T21:38:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:38:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:38:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[40587]: 8.b scrub starts 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[40587]: 8.b scrub ok 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[29406]: pgmap v432: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[29406]: 8.b scrub starts 2024-03-26T21:38:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:32 smithi055 ceph-mon[29406]: 8.b scrub ok 2024-03-26T21:38:33.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:33.103 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:33.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:33.237 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:32 smithi067 ceph-mon[33346]: pgmap v432: 148 pgs: 148 active+clean; 1.1 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-26T21:38:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:32 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:38:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:32 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:38:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:32 smithi067 ceph-mon[33346]: 8.b scrub starts 2024-03-26T21:38:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:32 smithi067 ceph-mon[33346]: 8.b scrub ok 2024-03-26T21:38:33.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:38:33.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:33.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:38:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.468 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:33.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:33.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:33.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:33.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:33.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:33.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:33.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:33.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:34.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:38:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 8.0 scrub starts 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[40587]: 8.0 scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:38:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 8.0 scrub starts 2024-03-26T21:38:34.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:38:34.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:38:34.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:33 smithi055 ceph-mon[29406]: 8.0 scrub ok 2024-03-26T21:38:34.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:34.146 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 8.0 scrub starts 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:38:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:38:34.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:33 smithi067 ceph-mon[33346]: 8.0 scrub ok 2024-03-26T21:38:34.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:34.287 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:34.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:34.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:34.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:34.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:34.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.837 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:34.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[40587]: pgmap v433: 148 pgs: 148 active+clean; 784 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[29406]: pgmap v433: 148 pgs: 148 active+clean; 784 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:38:34.893 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:34 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:38:34.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:34.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:35.010 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:35.010 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:35.072 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:35.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:35.132 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:38:35.181 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:38:35.204 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:35.220 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:38:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:34 smithi067 ceph-mon[33346]: pgmap v433: 148 pgs: 148 active+clean; 784 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:34 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:38:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:34 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:38:35.250 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:35.264 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:38:35.308 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:38:35.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:38:35.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:35.391 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:38:35.430 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:38:35.437 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:38:35.437 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:38:35.437 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:38:35.437 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:38:35.438 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:38:35.438 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:38:35.438 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:38:35.438 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:38:35.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.515 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:35.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:35.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:35.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:35.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:35.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:36.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:36.057 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[40587]: 8.2 scrub starts 2024-03-26T21:38:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:38:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[40587]: 8.2 scrub ok 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2475306523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[29406]: 8.2 scrub starts 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[29406]: 8.2 scrub ok 2024-03-26T21:38:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:35 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2475306523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:36.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:35 smithi067 ceph-mon[33346]: 8.2 scrub starts 2024-03-26T21:38:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:35 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:38:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:35 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:38:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:35 smithi067 ceph-mon[33346]: 8.2 scrub ok 2024-03-26T21:38:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:35 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2475306523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:36.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:36.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:36.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:36.608 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:36.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:36.749 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:36.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:36.888 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:36.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:36.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:37.011 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[29406]: pgmap v434: 148 pgs: 148 active+clean; 763 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[29406]: 8.1 scrub starts 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[40587]: pgmap v434: 148 pgs: 148 active+clean; 763 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[40587]: 8.1 scrub starts 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:38:37.012 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:36 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:38:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:37.122 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:37.158 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:36 smithi067 ceph-mon[33346]: pgmap v434: 148 pgs: 148 active+clean; 763 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T21:38:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:36 smithi067 ceph-mon[33346]: 8.1 scrub starts 2024-03-26T21:38:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:36 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:38:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:36 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:38:37.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:36 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:37.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:37.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:37.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:37.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:37.542 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:37.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:37.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:37.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:37.852 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:37.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:37.988 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:37.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:38.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:38.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: 8.1 scrub ok 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:38:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:37 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:38:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: 8.1 scrub ok 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: 8.1 scrub ok 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:38:38.255 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:38.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:38:38.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:37 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:38:38.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:38.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:38.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:38.622 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.660 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:38.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:38.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:38.792 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:38.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:38.938 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:38.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: pgmap v435: 148 pgs: 148 active+clean; 486 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: 8.7 scrub starts 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[40587]: 8.7 scrub ok 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: pgmap v435: 148 pgs: 148 active+clean; 486 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: 8.7 scrub starts 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:38:38.988 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:38:38.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-mon[29406]: 8.7 scrub ok 2024-03-26T21:38:39.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:39.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:38:39.080 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: pgmap v435: 148 pgs: 148 active+clean; 486 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: 8.7 scrub starts 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:38:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:38 smithi067 ceph-mon[33346]: 8.7 scrub ok 2024-03-26T21:38:39.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.226 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:39.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:39.254 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:39.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:39.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:39.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:39.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:39.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.676 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:39.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:39.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:39.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:39.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:39.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:40.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:40.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:40.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:40.264 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:40.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.410 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:40.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:40.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:40.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:40.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:40.807 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:40.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:40.948 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:40.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:41.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:41.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: pgmap v437: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T21:38:41.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:38:41.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: pgmap v437: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:38:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:40 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:38:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:41.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:41.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: pgmap v437: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:38:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:40 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:38:41.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:41.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:41.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:41.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:41.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:41.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:41.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:41.763 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:41.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:41.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:42.014 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:38:42.014 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:38:42.014 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:38:42.014 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:38:42.014 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:38:42.047 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:42.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:42.185 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 3.8 scrub starts 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 3.8 scrub ok 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:38:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:41 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 3.8 scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 3.8 scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:38:42.241 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 3.8 scrub starts 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 3.8 scrub ok 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:38:42.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:41 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:38:42.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:42.318 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:42.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:42.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:42.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:42.675 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Removed 113966 objects 2024-03-26T21:38:42.675 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Clean up completed and total clean up time :122.67 2024-03-26T21:38:42.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:42.720 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:42.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:42.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:42.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-26T21:38:42.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:42.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:43.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:43.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:43.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:43.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:42 smithi067 ceph-mon[33346]: pgmap v438: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:42 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:38:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:42 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:38:43.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[40587]: pgmap v438: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:43.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:38:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:38:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[29406]: pgmap v438: 148 pgs: 148 active+clean; 394 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T21:38:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:38:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:42 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:38:43.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:43.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:43.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:43.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:43.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:43.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:43.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:38:43.704 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:43.721 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:43.740 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:43.741 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:43.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:43.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:43.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:43.903 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:38:43.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.001 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:44.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:44.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.131 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:44.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:43 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:38:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:43 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:38:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/12279591' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:43 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:44.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:38:44.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/12279591' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/12279591' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:43 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:38:44.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:44.386 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:44.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:44.513 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:44.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:44.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:44.766 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:44.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:44.894 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:44.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: pgmap v439: 148 pgs: 148 active+clean; 61 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 8.a scrub starts 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 8.a scrub ok 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:38:44.969 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: pgmap v439: 148 pgs: 148 active+clean; 61 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 8.a scrub starts 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 8.a scrub ok 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:38:44.970 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:44 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:38:44.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:45.019 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:45.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:45.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: pgmap v439: 148 pgs: 148 active+clean; 61 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 8.a scrub starts 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 8.a scrub ok 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:38:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T21:38:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:38:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:38:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:38:45.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:44 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:38:45.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:45.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:45.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:45.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:45.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:45.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:45.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:45.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:45.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:45.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:45.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:46.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:46.034 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:46.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[29406]: 8.f scrub starts 2024-03-26T21:38:46.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[29406]: 8.f scrub ok 2024-03-26T21:38:46.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:38:46.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:38:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[40587]: 8.f scrub starts 2024-03-26T21:38:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[40587]: 8.f scrub ok 2024-03-26T21:38:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:38:46.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:45 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:38:46.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:46.163 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:45 smithi067 ceph-mon[33346]: 8.f scrub starts 2024-03-26T21:38:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:45 smithi067 ceph-mon[33346]: 8.f scrub ok 2024-03-26T21:38:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:45 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:38:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:45 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:38:46.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:46.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:46.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:46.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:46.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:46.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.676 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:46.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:46.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:46.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:46.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:46.930 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:46.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:47.032 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.061 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:47.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:47.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:47.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: pgmap v441: 148 pgs: 148 active+clean; 44 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 923 op/s 2024-03-26T21:38:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:38:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:47.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:47.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: pgmap v441: 148 pgs: 148 active+clean; 44 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 923 op/s 2024-03-26T21:38:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:38:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:38:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: pgmap v441: 148 pgs: 148 active+clean; 44 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 923 op/s 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:38:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:47.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:38:47.317 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:47.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:47.446 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:47.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:47.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:47.704 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:47.805 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:47.832 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:47.934 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:47.962 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:47.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:48.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:48.088 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:48.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 5.a deep-scrub starts 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 5.a deep-scrub ok 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:38:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 5.a deep-scrub starts 2024-03-26T21:38:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 5.a deep-scrub ok 2024-03-26T21:38:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:38:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:38:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 5.a deep-scrub starts 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 5.a deep-scrub ok 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:38:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:48.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:48.343 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:48.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:48.472 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:48.563 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-26T21:38:48.563 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-26T21:38:48.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:48.629 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:48.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:48.755 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:48.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:48.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:48.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 880 op/s 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 880 op/s 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:38:48.976 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:38:48.977 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:38:48.977 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:38:48.977 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:38:48.977 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:38:48.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:49.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:49.133 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:49.228 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:49.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 880 op/s 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:38:49.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:48 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:38:49.261 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:49.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:49.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:49.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:49.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:49.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:49.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:49.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:49.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:49.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:49.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:49.913 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:49.935 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:38:49.958 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:38:49.977 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:38:49.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:49.997 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:38:50.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:50.020 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:38:50.040 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:38:50.061 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:38:50.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[29406]: 6.17 deep-scrub starts 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[29406]: 6.17 deep-scrub ok 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[40587]: 6.17 deep-scrub starts 2024-03-26T21:38:50.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:49 smithi055 ceph-mon[40587]: 6.17 deep-scrub ok 2024-03-26T21:38:50.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:38:50.083 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:38:50.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:49 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:38:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:49 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:38:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:49 smithi067 ceph-mon[33346]: 6.17 deep-scrub starts 2024-03-26T21:38:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:49 smithi067 ceph-mon[33346]: 6.17 deep-scrub ok 2024-03-26T21:38:50.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:50.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:50.374 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:50.400 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:50.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:50.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:50.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:50.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:50.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:50.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:50.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:51.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:51.038 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/596201443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:38:51.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/596201443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:38:51.125 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:50 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:38:51.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:51.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:38:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.164 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:51.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/596201443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:38:51.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:50 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:38:51.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:51.292 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:51.394 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.418 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:51.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:51.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:51.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:51.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:51.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:51.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:51.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:51.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:52.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:52.054 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:52.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:52.183 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:38:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:51 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:38:52.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:38:52.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:38:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:51 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:38:52.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:52.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:52.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:52.559 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:52.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:52.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:52.811 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:52.851 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:38:52.911 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:52.911 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:52.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:52.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:52.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:52.945 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:52.945 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:53.043 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:38:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:53.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:53.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:52 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:53.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:53.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:53.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:38:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:38:53.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:38:53.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:38:53.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:52 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:38:53.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:53.318 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:53.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:53.444 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:53.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-26T21:38:53.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:53.569 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:53.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:53.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:53.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:53.821 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:53.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:53.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:53.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:54.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:54.071 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:54.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:54.197 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 5.1 deep-scrub starts 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 5.1 deep-scrub ok 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 3.16 deep-scrub starts 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 3.16 deep-scrub ok 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:38:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:53 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:38:54.297 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 5.1 deep-scrub starts 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 5.1 deep-scrub ok 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 3.16 deep-scrub starts 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 3.16 deep-scrub ok 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 5.1 deep-scrub starts 2024-03-26T21:38:54.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 5.1 deep-scrub ok 2024-03-26T21:38:54.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 3.16 deep-scrub starts 2024-03-26T21:38:54.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 3.16 deep-scrub ok 2024-03-26T21:38:54.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:38:54.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:53 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:38:54.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:54.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:54.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:38:54.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:54.449 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:54.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:54.570 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:38:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:54.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:54.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:54.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:54.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:54.951 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:54.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: pgmap v445: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 155 op/s 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 8.5 scrub starts 2024-03-26T21:38:54.960 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[29406]: 8.5 scrub ok 2024-03-26T21:38:55.053 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.077 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:55.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:55.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:55.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: pgmap v445: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 155 op/s 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 8.5 scrub starts 2024-03-26T21:38:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:54 smithi067 ceph-mon[33346]: 8.5 scrub ok 2024-03-26T21:38:55.307 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: pgmap v445: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 155 op/s 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 8.5 scrub starts 2024-03-26T21:38:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:54 smithi055 ceph-mon[40587]: 8.5 scrub ok 2024-03-26T21:38:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:55.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:55.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:55.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:55.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:55.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:55.711 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:55.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:55.836 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:55.937 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:55.963 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:55.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:38:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:55 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:38:56.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:56.166 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:38:56.166 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:38:56.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:38:56.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:38:56.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:38:56.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:38:56.167 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:38:56.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:38:56.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:38:56.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:38:56.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:38:56.168 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:55 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:38:56.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:56.311 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:56.311 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:56.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:56.341 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:56.422 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:38:56.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:56.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:38:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:56.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:38:56.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:56.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:56.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:56.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:56.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:56.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:56.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:56.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:38:57.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:57.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:57.176 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[40587]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 95 op/s 2024-03-26T21:38:57.176 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:38:57.176 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:38:57.176 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[40587]: 8.9 scrub starts 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[40587]: 8.9 scrub ok 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[29406]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 95 op/s 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[29406]: 8.9 scrub starts 2024-03-26T21:38:57.177 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:56 smithi055 ceph-mon[29406]: 8.9 scrub ok 2024-03-26T21:38:57.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:57.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:57.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:57.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:38:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:56 smithi067 ceph-mon[33346]: pgmap v446: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 95 op/s 2024-03-26T21:38:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:56 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:38:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:56 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:38:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:56 smithi067 ceph-mon[33346]: 8.9 scrub starts 2024-03-26T21:38:57.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:56 smithi067 ceph-mon[33346]: 8.9 scrub ok 2024-03-26T21:38:57.246 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:38:57.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:57.350 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:57.452 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:57.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:57.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:57.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:57.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:38:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:38:57.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:57.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:57.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:57.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[40587]: 8.3 scrub starts 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[40587]: 8.3 scrub ok 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[29406]: 8.3 scrub starts 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[29406]: 8.3 scrub ok 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:38:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:57 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:38:58.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:58.103 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:58.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:58.221 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-mon[33346]: 8.3 scrub starts 2024-03-26T21:38:58.221 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-mon[33346]: 8.3 scrub ok 2024-03-26T21:38:58.221 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:38:58.221 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:38:58.221 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:57 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:38:58.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:58.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:38:58.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:38:58.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:58.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:58.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:58.603 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:58.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:58.728 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:38:58.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:58.853 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:58.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:58.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:58.988 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_1' removed 2024-03-26T21:38:59.007 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:38:59.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:59.114 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:59.136 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:38:59.136 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:38:59.136 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-26T21:38:59.136 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:38:59.136 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:38:59.136 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 5.d deep-scrub starts 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 5.d deep-scrub ok 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 2.2 deep-scrub starts 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 2.2 deep-scrub ok 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/28325594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:38:59.137 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:38:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:38:59.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:59.243 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:38:59 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:38:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 5.d deep-scrub starts 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 5.d deep-scrub ok 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 2.2 deep-scrub starts 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 2.2 deep-scrub ok 2024-03-26T21:38:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/28325594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: pgmap v447: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 5.d deep-scrub starts 2024-03-26T21:38:59.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 5.d deep-scrub ok 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 2.2 deep-scrub starts 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 2.2 deep-scrub ok 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/28325594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:38:59.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:38:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:38:59.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:59.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:59.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:38:59.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:38:59.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:38:59.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:38:59.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:38:59.746 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:38:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:38:59.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:38:59.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:38:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:38:59.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:38:59.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:38:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:38:59.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:00.035 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:00.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:00.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:00.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:00.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:39:00.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:39:00.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:00.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:39:00.242 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:00 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:39:00.250 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2437720269' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:39:00.269 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:00 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:39:00.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:00.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:00.477 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:00.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:00.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:00.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:00.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:00.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:00.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:00.872 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T21:39:00.873 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T21:39:00.874 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-26T21:39:00.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-26T21:39:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:00.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:00.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:00.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:01.082 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:01.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:01.133 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:01.178 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:01 smithi067 ceph-mon[33346]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:01.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:01 smithi055 ceph-mon[40587]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:01.208 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:01 smithi055 ceph-mon[29406]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:01.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:01.259 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:01.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:01.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:01.510 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:01.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:01.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:01.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-26T21:39:01.733 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:01.733 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:01.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:01.740 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:39:01.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:01.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:01.899 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:01.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:02.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:02.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:02.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:02.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:02.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:02.276 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:02.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:02.373 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:02 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:02.373 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:02 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:02.373 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:02.374 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:02.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:02.397 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:02.406 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:02.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:39:02.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:02.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:02.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:02.659 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:02.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:02.786 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:02.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:02.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:03.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:03.040 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:03.041 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:03.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:03.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:03 smithi067 ceph-mon[33346]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:03.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:03 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:03.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:03 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:03.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:03 smithi067 ceph-mon[33346]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:39:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:03.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:03.279 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[29406]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:03.279 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:03.279 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:03.279 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[29406]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:39:03.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[40587]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-03-26T21:39:03.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:03.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:03.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:03 smithi055 ceph-mon[40587]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T21:39:03.293 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:03.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:03.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:03.534 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:39:03.544 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:03.555 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:39:03.575 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:39:03.595 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:39:03.615 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:39:03.635 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:39:03.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:03.656 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:39:03.669 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:03.676 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:39:03.677 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:39:03.678 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:39:03.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:03.794 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:03.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:03.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:03.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:04.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:04.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:04.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:04.174 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2924199138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:04.275 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:04.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2924199138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:04.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2924199138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:04.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:04.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:04.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:39:04.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:39:04.532 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-26T21:39:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:39:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:04.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:04.605 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:04.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:04.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:04.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:04.924 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:04.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:05.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:05.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:05.176 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:05.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:05 smithi067 ceph-mon[33346]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:05.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:39:05.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:05 smithi055 ceph-mon[29406]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:05.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:05 smithi055 ceph-mon[40587]: pgmap v452: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:05.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:05.489 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:05.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:05.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:05.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:05.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:05.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:05.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:05.942 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:05.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:06.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:06.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:06.171 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.196 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:06.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:06.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:06.321 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:06.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:06.447 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:06.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:06.573 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:06.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:06.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:06.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:06.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:06.952 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:06.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:07.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:07.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:07.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:07.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:07.218 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:39:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:39:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:39:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:39:07.219 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:39:07.219 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:07 smithi067 ceph-mon[33346]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:39:07.219 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:07.227 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:07 smithi055 ceph-mon[29406]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:39:07.227 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:07 smithi055 ceph-mon[40587]: pgmap v453: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:39:07.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:07.328 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:07.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:07.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:07.580 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:07.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:07.706 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:07.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:07.831 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:07.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:07.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:39:07.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:07.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:08.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:08.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:08.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:08.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:08 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:08 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:08.245 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:08.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:08.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:08.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:08.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:08.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:39:08.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:08.592 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:08.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:08.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:08.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:08.967 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:08.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:09.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:09.094 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:09.095 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:39:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:09.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:09.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[40587]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:09.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[40587]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:39:09.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/45467556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.273 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.274 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[29406]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:09.274 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[29406]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:39:09.274 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/45467556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.274 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:09 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.274 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:09.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:09.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:09.373 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:09 smithi067 ceph-mon[33346]: pgmap v454: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:39:09.374 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:09 smithi067 ceph-mon[33346]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T21:39:09.374 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:09 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/45467556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.374 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:09 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:09.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-26T21:39:09.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:09.507 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:09.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:09.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:09.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:09.758 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:09.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:09.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:09.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:39:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:39:09.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1123 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94208173341440 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 519.19376080799998, 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0115813600000001, 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:39:09.921 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1123 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732482+0000", 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732483+0000", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2589999999999999e-06 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732488+0000", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7539999999999997e-06 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732499+0000", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1243000000000001e-05 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732508+0000", 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0040000000000005e-06 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732567+0000", 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8811000000000003e-05 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732573+0000", 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2330000000000003e-06 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736258+0000", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036856459999999999 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.736267+0000", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7460000000000006e-06 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739066+0000", 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.002799072 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.739206+0000", 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013932300000000001 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.744063+0000", 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004857269 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:1124 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94208178013312 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 519.19359857500001, 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.011248098, 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1124 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732644+0000", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4099999999999998e-07 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732648+0000", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.771e-06 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732657+0000", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9639999999999992e-06 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732679+0000", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1985999999999999e-05 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732893+0000", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021387399999999999 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:30.732903+0000", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9499999999999996e-06 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738098+0000", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.005195499 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.738271+0000", 2024-03-26T21:39:09.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172795 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:30:31.743892+0000", 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005620618 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197497 7.d 7:ba42a995:::benchmark_data_smithi055_81491_object78010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 321.18903110999997, 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.166716791, 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197497 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737211+0000", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737213+0000", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3599999999999999e-06 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737222+0000", 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.842e-06 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737236+0000", 2024-03-26T21:39:09.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3692e-05 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737244+0000", 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3989999999999998e-06 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737302+0000", 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8579000000000003e-05 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737433+0000", 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013079199999999999 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,1", 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.737563+0000", 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000129846 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.740324+0000", 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027609940000000001 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.741152+0000", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082808100000000004 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903860+0000", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.162708202 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903889+0000", 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9422999999999999e-05 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903928+0000", 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8581000000000002e-05 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197503 7.7 7:e387ae8a:::benchmark_data_smithi055_81491_object78016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 321.18260359599998, 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15943073799999999, 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197503 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743639+0000", 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743640+0000", 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.23e-07 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743644+0000", 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4279999999999998e-06 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743653+0000", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4429999999999992e-06 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743661+0000", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6090000000000003e-06 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743706+0000", 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4637e-05 2024-03-26T21:39:09.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743794+0000", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8764999999999995e-05 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.743873+0000", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8942000000000004e-05 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.746188+0000", 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023149080000000001 2024-03-26T21:39:09.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.895858+0000", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14966966300000001 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.902961+0000", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071028239999999998 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903017+0000", 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6657000000000003e-05 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.903069+0000", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2039000000000001e-05 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84024 8.5 8:a9a6864a:::benchmark_data_smithi055_110126_object84023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.394747712, 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18444987099999999, 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84024 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531496+0000", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.73e-06 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531543+0000", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6681999999999998e-05 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531556+0000", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3032e-05 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531564+0000", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1289999999999996e-06 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555104+0000", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023539966999999998 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555293+0000", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018880700000000001 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555499+0000", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020556200000000001 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564809+0000", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093100309999999999 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567429+0000", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002620645 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712266+0000", 2024-03-26T21:39:09.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14483680500000001 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712311+0000", 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4746000000000002e-05 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.715945+0000", 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036337349999999999 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84035 8.5 8:a52a9b25:::benchmark_data_smithi055_110126_object84034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.381956473, 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.17997925100000001, 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84035 2024-03-26T21:39:09.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544286+0000", 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544287+0000", 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4079999999999999e-06 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544397+0000", 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010953 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544421+0000", 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4443999999999998e-05 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544429+0000", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6680000000000001e-06 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558408+0000", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013978945 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558584+0000", 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017627100000000001 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.558791+0000", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020727899999999999 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567383+0000", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085917289999999993 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567490+0000", 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010709000000000001 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.720857+0000", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15336719900000001 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.720902+0000", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4885000000000001e-05 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724265+0000", 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033628030000000001 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84036 8.5 8:a733162b:::benchmark_data_smithi055_110126_object84035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.38179624899999, 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18335849000000001, 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84036 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544447+0000", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.78e-07 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544508+0000", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0568e-05 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544530+0000", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2268000000000001e-05 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544536+0000", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2829999999999998e-06 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564845+0000", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020308641999999998 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565017+0000", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017215199999999999 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565220+0000", 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020294399999999999 2024-03-26T21:39:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576111+0000", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010890982 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576274+0000", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000162819 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724342+0000", 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14806820500000001 2024-03-26T21:39:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724408+0000", 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6094999999999997e-05 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727805+0000", 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033966539999999998 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84038 8.5 8:a3a8abbc:::benchmark_data_smithi055_110126_object84037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.38169753099999, 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18689033499999999, 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84038 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:39:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.06e-07 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544585+0000", 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9360000000000003e-05 2024-03-26T21:39:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544592+0000", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-06 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544598+0000", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5310000000000001e-06 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567532+0000", 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022934059 2024-03-26T21:39:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567708+0000", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017579499999999999 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567936+0000", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000228173 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576172+0000", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082357030000000005 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576281+0000", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109747 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727868+0000", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.151586689 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727982+0000", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011343700000000001 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731435+0000", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003453631 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84039 8.5 8:a8845c01:::benchmark_data_smithi055_110126_object84038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.38163573899999, 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.190558218, 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.967 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84039 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4400000000000001e-07 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544631+0000", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.442e-05 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544638+0000", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3790000000000003e-06 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544643+0000", 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7160000000000002e-06 2024-03-26T21:39:09.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570111+0000", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025468094 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570292+0000", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018180200000000001 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570501+0000", 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208727 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576257+0000", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057555239999999997 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584951+0000", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086946679999999991 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731584+0000", 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.146632868 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731658+0000", 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3287999999999994e-05 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735165+0000", 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035072879999999999 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84048 8.5 8:a98c3a1d:::benchmark_data_smithi055_110126_object84047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.370170634, 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.182857732, 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84048 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556073+0000", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.584e-06 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556178+0000", 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104714 2024-03-26T21:39:09.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556189+0000", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0582e-05 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556196+0000", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8259999999999993e-06 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573081+0000", 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016884952000000002 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573261+0000", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017950600000000001 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573469+0000", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020786200000000001 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584956+0000", 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011487264000000001 2024-03-26T21:39:09.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585011+0000", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4629000000000002e-05 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735223+0000", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.150211927 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735322+0000", 2024-03-26T21:39:09.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9017000000000004e-05 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.738929+0000", 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036078690000000001 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84053 8.5 8:a46639cb:::benchmark_data_smithi055_110126_object84052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:39:09.978 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 187.36226973999999, 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.17847104499999999, 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84053 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563973+0000", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.563974+0000", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4449999999999999e-06 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564042+0000", 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8171999999999997e-05 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564054+0000", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1537e-05 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564062+0000", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2740000000000007e-06 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576328+0000", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012265728 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576475+0000", 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014767799999999999 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576658+0000", 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018290400000000001 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585057+0000", 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0083985689999999998 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585066+0000", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9139999999999997e-06 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.738983+0000", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15391669699999999 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.739084+0000", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010157000000000001 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.742444+0000", 2024-03-26T21:39:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003359557 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84738 8.5 8:ad591b1d:::benchmark_data_smithi055_110126_object84737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.32518651000001, 2024-03-26T21:39:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.165289678, 2024-03-26T21:39:09.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84738 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601056+0000", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601057+0000", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.437e-06 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601116+0000", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9123000000000003e-05 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601126+0000", 2024-03-26T21:39:09.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3039999999999994e-06 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.601132+0000", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5939999999999999e-06 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.611839+0000", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010707003999999999 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.611997+0000", 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157935 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.612193+0000", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196219 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.628571+0000", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016377571 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633071+0000", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045002790000000003 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.763063+0000", 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12999149400000001 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.763143+0000", 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0453999999999999e-05 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766346+0000", 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032022639999999998 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84759 8.5 8:ab1dad65:::benchmark_data_smithi055_110126_object84758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.30732796199999, 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14978145100000001, 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84759 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618914+0000", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618916+0000", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7349999999999999e-06 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618958+0000", 2024-03-26T21:39:09.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1878000000000001e-05 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618969+0000", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.084e-05 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.618977+0000", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8890000000000002e-06 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625426+0000", 2024-03-26T21:39:09.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006449092 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625618+0000", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019185299999999999 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.625829+0000", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000210965 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633120+0000", 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072909649999999999 2024-03-26T21:39:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640696+0000", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075759269999999997 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766432+0000", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.125736666 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.766478+0000", 2024-03-26T21:39:09.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5377999999999998e-05 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768696+0000", 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022182629999999998 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84763 8.5 8:ae380329:::benchmark_data_smithi055_110126_object84762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.302815899, 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.147994242, 2024-03-26T21:39:09.996 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84763 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623426+0000", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623435+0000", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0280000000000006e-06 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623469+0000", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3723000000000002e-05 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623484+0000", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4858000000000001e-05 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.623492+0000", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6879999999999999e-06 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633153+0000", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096608590000000008 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633302+0000", 2024-03-26T21:39:09.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149326 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.633480+0000", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017833700000000001 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640675+0000", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071944629999999999 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640704+0000", 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-03-26T21:39:10.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768765+0000", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12806050699999999 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.768831+0000", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6161999999999994e-05 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.771421+0000", 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025899830000000001 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84768 8.5 8:afce3940:::benchmark_data_smithi055_110126_object84767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.296426102, 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.166609485, 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.002 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84768 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629816+0000", 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629818+0000", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7179999999999999e-06 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629872+0000", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4032e-05 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629886+0000", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.431e-05 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.629894+0000", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0139999999999998e-06 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637716+0000", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078216980000000002 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637805+0000", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9189000000000003e-05 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.637951+0000", 2024-03-26T21:39:10.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014564800000000001 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.640755+0000", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028039010000000001 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.641376+0000", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062082399999999996 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.792216+0000", 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.150839943 2024-03-26T21:39:10.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.792277+0000", 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.19e-05 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:03.796426+0000", 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041483079999999999 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:86639 8.5 8:a036fec7:::benchmark_data_smithi055_110126_object86638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:10.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 183.89666985100001, 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.236906851, 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 86639 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:39:10.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029574+0000", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.271e-06 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029628+0000", 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4159000000000001e-05 2024-03-26T21:39:10.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029639+0000", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0638e-05 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029646+0000", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1629999999999999e-06 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.036918+0000", 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072720809999999997 2024-03-26T21:39:10.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037144+0000", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022611500000000001 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037343+0000", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019870900000000001 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.040140+0000", 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00279715 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.042107+0000", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019675669999999999 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262797+0000", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.220689684 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262862+0000", 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5030000000000006e-05 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.266479+0000", 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036172840000000001 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:91648 8.5 8:a0c9058a:::benchmark_data_smithi055_110126_object91647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:11.861223+0000", 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 178.065018944, 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.149250101, 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.013 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 91648 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861223+0000", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861223+0000", 2024-03-26T21:39:10.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861225+0000", 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5340000000000001e-06 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861264+0000", 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9344999999999997e-05 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861277+0000", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2285e-05 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.861287+0000", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0710000000000001e-05 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.868495+0000", 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072081979999999999 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.868665+0000", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169844 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.868905+0000", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240096 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.871909+0000", 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030040179999999998 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:11.871956+0000", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6317999999999999e-05 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:12.007897+0000", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13594157800000001 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:12.007939+0000", 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1470000000000001e-05 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:12.010474+0000", 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025347049999999999 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108533 8.5 8:a9c0f906:::benchmark_data_smithi055_110126_object108532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 158.673550272, 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21060493199999999, 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.019 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108533 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252694+0000", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5960000000000001e-06 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252736+0000", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2651999999999998e-05 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252750+0000", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3966e-05 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252759+0000", 2024-03-26T21:39:10.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271069+0000", 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018310448999999999 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271232+0000", 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016281499999999999 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271366+0000", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013423299999999999 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275699+0000", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043322029999999997 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369423+0000", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093724093999999994 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.460020+0000", 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090597103999999998 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.460100+0000", 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0211999999999997e-05 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463297+0000", 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031969149999999998 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108536 8.5 8:adbf2f7d:::benchmark_data_smithi055_110126_object108535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 158.670943021, 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21185876100000001, 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108536 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255300+0000", 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4900000000000005e-07 2024-03-26T21:39:10.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255347+0000", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6761000000000002e-05 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255357+0000", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0264e-05 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255364+0000", 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7000000000000002e-06 2024-03-26T21:39:10.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273327+0000", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017963257999999999 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273484+0000", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015665699999999999 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273673+0000", 2024-03-26T21:39:10.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189648 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.277642+0000", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039683460000000002 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369476+0000", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091834425999999997 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463380+0000", 2024-03-26T21:39:10.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093904038999999995 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463452+0000", 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1627999999999999e-05 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467158+0000", 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037061849999999999 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108537 8.5 8:a9ba36a4:::benchmark_data_smithi055_110126_object108536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 158.670790203, 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21534863500000001, 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108537 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.031 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255458+0000", 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1469999999999998e-06 2024-03-26T21:39:10.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255515+0000", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6891000000000001e-05 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255524+0000", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5019999999999992e-06 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255529+0000", 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.186e-06 2024-03-26T21:39:10.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275731+0000", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02020183 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275839+0000", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010788 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275964+0000", 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012569299999999999 2024-03-26T21:39:10.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.277748+0000", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001783506 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369518+0000", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091770298 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467221+0000", 2024-03-26T21:39:10.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097703264999999997 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467338+0000", 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116978 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.470801+0000", 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034624590000000002 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:10.036 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:39:10.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:10.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:10 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:10 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:10 smithi067 ceph-mon[33346]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:39:10.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:10.298 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:10.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:10.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[29406]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:39:10.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:10.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:10.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:10 smithi055 ceph-mon[40587]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T21:39:10.317 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:10.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:10.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:10.530 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:10.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:10.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:10.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:10.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:10.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:39:10.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:10.822 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:10.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:10.947 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:10.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:11.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:11.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:11.123 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_2' created 2024-03-26T21:39:11.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:11.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:11.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:11.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[40587]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-26T21:39:11.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2458281873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.303 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.303 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[29406]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-26T21:39:11.303 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2458281873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.303 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.322 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:11.369 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:11 smithi067 ceph-mon[33346]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-03-26T21:39:11.369 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:11 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2458281873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.369 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T21:39:11.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:11.447 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:11.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:11.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:11.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:11.824 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:11.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:11.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:11.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:11.954 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:12.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:12.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:39:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:12.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:12.303 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:12.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:12 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T21:39:12.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:12 smithi055 ceph-mon[40587]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:39:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:12 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T21:39:12.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:12 smithi055 ceph-mon[29406]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:39:12.326 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:12.375 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:12 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T21:39:12.375 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:12 smithi067 ceph-mon[33346]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T21:39:12.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:12.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-26T21:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:12.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:12.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:12.762 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:39:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:39:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-26T21:39:12.763 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:39:12.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:12.834 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:12.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:12.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:12.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:13.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:13.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:13.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:13.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:13.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[40587]: pgmap v459: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:39:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[40587]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:39:13.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[29406]: pgmap v459: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:13.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:39:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:13 smithi055 ceph-mon[29406]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:39:13.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:13.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:13.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:13 smithi067 ceph-mon[33346]: pgmap v459: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:13 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:39:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:13 smithi067 ceph-mon[33346]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T21:39:13.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:13.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:39:13.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:13.717 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:13.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:13.723 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:13.759 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:39:13.780 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:39:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:39:13.820 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:39:13.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:13.839 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:39:13.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:13.860 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:39:13.880 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:39:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:13.901 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:39:13.902 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:39:13.902 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:39:13.902 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:39:13.902 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:39:13.903 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:39:13.903 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:39:13.903 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:39:13.903 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:39:13.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:13.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:13.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:14.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:14.098 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:14.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:14.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:14.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:14 smithi055 ceph-mon[29406]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:39:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/568215139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:14 smithi055 ceph-mon[40587]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:39:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/568215139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:14.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:14.343 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:14 smithi067 ceph-mon[33346]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T21:39:14.343 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/568215139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:14.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:14.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:14.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:14.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:14.704 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:14.730 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:14.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:14.854 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:14.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:14.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:14.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:15.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.111 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:15.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:15.235 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:15.336 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:15.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:15.450 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:15 smithi067 ceph-mon[33346]: pgmap v462: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:15.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:15.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:15 smithi055 ceph-mon[40587]: pgmap v462: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:15.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:15 smithi055 ceph-mon[29406]: pgmap v462: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:39:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:15.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.613 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:15.706 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:15.706 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:15.714 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:15.737 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:15.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:15.862 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:15.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:15.988 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:15.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:16.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:16.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:16.240 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:16.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:16.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:39:16.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.493 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:16.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:16.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:16.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:16.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:16.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:16.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:16.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:16.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:16.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:17.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:17.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:17.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:17.165 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[29406]: pgmap v463: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:17.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:17.249 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:17.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:17.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:17.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:17 smithi067 ceph-mon[33346]: pgmap v463: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:17.500 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[40587]: pgmap v463: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:39:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:17.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:17.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:17.750 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:17.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:17.876 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:17.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:17.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:18.003 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:18.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:18.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:18.176 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:39:18.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:18.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:18.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:18.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:18.430 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:18.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:39:18.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:18 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:18.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:18 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:18.431 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:18.431 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:18.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:18.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:39:18.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:18.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:18.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:18.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:39:18.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:18.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:18.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:18.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:18.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:18.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.756 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:18.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:18.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:18.881 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:18.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:18.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:19.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:39:19.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:19.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:19.143 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:19.185 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-26T21:39:19.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:19.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:19.319 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:19.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2983942062' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:39:19.436 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:19.437 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:39:19.437 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:39:19.437 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2983942062' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.437 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:19 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1357953746' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2983942062' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:19 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:39:19.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:19.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:19.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:19.769 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:19.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:19.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:19.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:20.021 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:20.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-26T21:39:20.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:20.163 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:39:20.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:39:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:39:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:39:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[40587]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:39:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:39:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:39:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:39:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:20 smithi055 ceph-mon[29406]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:39:20.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:20 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:39:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:20 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:39:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:20 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:39:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:20 smithi067 ceph-mon[33346]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T21:39:20.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:20.522 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:20.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:20.648 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:20.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:20.773 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:20.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:20.902 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:20.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:20.959 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:20.982 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:21.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:21.028 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:21.057 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:21.057 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:21.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:21.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:21.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:21.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.404 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:21.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:39:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:21 smithi067 ceph-mon[33346]: pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-03-26T21:39:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:21 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:39:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:21 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:39:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:21 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:39:21.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[40587]: pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[29406]: pgmap v467: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:39:21.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:21 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:39:21.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:21.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:21.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:21.655 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:21.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:39:21.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:21.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:21.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:21.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:21.932 DEBUG:teuthology.orchestra.run.smithi055:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-26T21:39:21.934 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T21:39:21.935 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T21:39:21.983 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:hints = 1 2024-03-26T21:39:21.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T21:39:21.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object prefix: benchmark_data_smithi055_135863 2024-03-26T21:39:21.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:39:21.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 0 0 0 0 0 0 - 0 2024-03-26T21:39:22.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:22.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:22.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:22.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.336 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:22.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:22.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:22.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:22.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:22.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:22.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:22.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:22.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:22.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 1 15 3535 3520 219.988 220 0.00386059 0.00453061 2024-03-26T21:39:22.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:23.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:23.161 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:23.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:23.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:23.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:23 smithi067 ceph-mon[33346]: pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:39:23.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:23 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:39:23.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:23 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:39:23.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:23.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:23.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[40587]: pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:39:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:39:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:39:23.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[29406]: pgmap v468: 148 pgs: 148 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:39:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:39:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:23 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:39:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:23.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:23.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:23.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:23.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:23.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:23.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 2 16 6791 6775 211.704 203.438 0.00638868 0.00471408 2024-03-26T21:39:24.009 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:24.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:24.123 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:24.123 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:24.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:24.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:39:24.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:39:24.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:39:24.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:39:24.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:39:24.248 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:39:24.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:39:24.249 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:24 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:39:24.279 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:24.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.422 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:24.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:39:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:24 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:39:24.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:24.564 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:24.607 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:39:24.656 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:39:24.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:24.691 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:24.698 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:39:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:39:24.784 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:39:24.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:24.826 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:39:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:24.871 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:39:24.910 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:39:24.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:39:24.939 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:24.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:24.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:24.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 3 16 9364 9348 194.735 160.812 0.00439031 0.00512719 2024-03-26T21:39:25.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:25.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:25.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:39:25.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:25.261 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:25.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:25.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: pgmap v469: 148 pgs: 148 active+clean; 75 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:39:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:39:25.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:25 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2721540942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:25.503 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:25.534 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: pgmap v469: 148 pgs: 148 active+clean; 75 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:39:25.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2721540942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:25.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: pgmap v469: 148 pgs: 148 active+clean; 75 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:25.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:39:25.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:39:25.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:39:25.537 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:25 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2721540942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:25.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:25.585 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:39:25.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:25.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:25.833 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:25.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:25.974 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:25.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:25.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 4 16 12350 12334 192.704 186.625 0.00602596 0.00518164 2024-03-26T21:39:26.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:26.116 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:26.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:26.258 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:26.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:26.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:26 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:39:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:26 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:39:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:26 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:39:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:26 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:39:26.502 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:26.541 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:39:26.544 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:39:26.545 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:26 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:39:26.643 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:26.682 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:26.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:26.818 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:26.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:26.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:26.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:26.984 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 5 16 15106 15090 188.61 172.25 0.00720345 0.0052954 2024-03-26T21:39:27.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:27.084 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:27.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:27.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:27.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:27.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:27.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:27.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-mon[33346]: pgmap v470: 148 pgs: 148 active+clean; 75 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:27.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:39:27.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:39:27.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:39:27.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:27 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:39:27.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-26T21:39:27.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[40587]: pgmap v470: 148 pgs: 148 active+clean; 75 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[29406]: pgmap v470: 148 pgs: 148 active+clean; 75 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 9.3 MiB/s wr, 149 op/s 2024-03-26T21:39:27.479 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:39:27.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:39:27.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:39:27.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:27 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:39:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:27.599 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151347 7.1 7.a9a74ce1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 393.11866157999998, 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672471500000001, 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151347 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484849+0000", 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484851+0000", 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6699999999999998e-06 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484856+0000", 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7890000000000002e-06 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484864+0000", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9589999999999995e-06 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484873+0000", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1759999999999999e-06 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506221+0000", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021347940999999999 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506230+0000", 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0170000000000002e-06 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451562+0000", 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.945331378 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451574+0000", 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1785e-05 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151351 7.1 7.ac3a4ac1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 393.11862571, 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672367400000003, 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151351 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484885+0000", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484886+0000", 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5069999999999999e-06 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484889+0000", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2720000000000002e-06 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484895+0000", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8239999999999996e-06 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484901+0000", 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7030000000000003e-06 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506254+0000", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021352684 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506260+0000", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8440000000000003e-06 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451599+0000", 2024-03-26T21:39:27.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94533952600000004 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451608+0000", 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3139999999999993e-06 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151354 7.1 7.e5ae2af1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 393.118601587, 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96672899499999998, 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.609 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151354 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484909+0000", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484911+0000", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.17e-06 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484916+0000", 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6129999999999997e-06 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484920+0000", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9639999999999997e-06 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484926+0000", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000002e-06 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506282+0000", 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021356001999999999 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506287+0000", 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6210000000000001e-06 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451630+0000", 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94534287400000006 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:27.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451638+0000", 2024-03-26T21:39:27.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6709999999999999e-06 2024-03-26T21:39:27.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.614 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:151355 7.1 7.e69c8861 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 393.11857784, 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96673204199999996, 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 151355 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484933+0000", 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484935+0000", 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0640000000000001e-06 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484938+0000", 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4010000000000001e-06 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484942+0000", 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.459e-06 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.484947+0000", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9609999999999996e-06 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506313+0000", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021365129 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:54.506319+0000", 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.409e-06 2024-03-26T21:39:27.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451657+0000", 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94533827999999998 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:32:55.451665+0000", 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3390000000000004e-06 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197517 7.1 7:8675bb79:::benchmark_data_smithi055_81491_object78030:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.619 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.913860+0000", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.68965055500001, 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3876742740000001, 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197517 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913860+0000", 2024-03-26T21:39:27.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913860+0000", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913861+0000", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0720000000000001e-06 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913865+0000", 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2719999999999999e-06 2024-03-26T21:39:27.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913875+0000", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9839999999999996e-06 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913883+0000", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8639999999999996e-06 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913919+0000", 2024-03-26T21:39:27.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6430999999999997e-05 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913925+0000", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6969999999999998e-06 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.294499+0000", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3805740289999999 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.294591+0000", 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2311000000000004e-05 2024-03-26T21:39:27.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301487+0000", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00689545 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301534+0000", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7163999999999998e-05 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197518 7.1 7:8d2c109a:::benchmark_data_smithi055_81491_object78031:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.913901+0000", 2024-03-26T21:39:27.624 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.68960890599999, 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3995876869999999, 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197518 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913901+0000", 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913901+0000", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913902+0000", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-07 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913905+0000", 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1590000000000002e-06 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913912+0000", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2119999999999997e-06 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913921+0000", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5050000000000007e-06 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913945+0000", 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4062000000000001e-05 2024-03-26T21:39:27.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.913948+0000", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3009999999999998e-06 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.294822+0000", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3808738039999999 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.294878+0000", 2024-03-26T21:39:27.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5716e-05 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313396+0000", 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018518435999999999 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313489+0000", 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2979000000000005e-05 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197529 7.1 7:8fe4cab8:::benchmark_data_smithi055_81491_object78042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.918680+0000", 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.684830197, 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.382668939, 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197529 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.630 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918680+0000", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918680+0000", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918682+0000", 2024-03-26T21:39:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4529999999999999e-06 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918687+0000", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4070000000000001e-06 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918699+0000", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2357999999999999e-05 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918711+0000", 2024-03-26T21:39:27.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1744e-05 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918761+0000", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0306000000000003e-05 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.918770+0000", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0189999999999995e-06 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.294916+0000", 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3761459890000001 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.295000+0000", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4012999999999997e-05 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.295082+0000", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1885999999999995e-05 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.299946+0000", 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048634100000000003 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301053+0000", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001106852 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301192+0000", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013954700000000001 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301250+0000", 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8189000000000001e-05 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301349+0000", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8765999999999996e-05 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197537 7.1 7:8ae4a579:::benchmark_data_smithi055_81491_object78050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.923411+0000", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.68009942600003, 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3783895420000001, 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197537 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923411+0000", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923411+0000", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923412+0000", 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1710000000000001e-06 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923417+0000", 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6240000000000001e-06 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923427+0000", 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0592e-05 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923434+0000", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1820000000000003e-06 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923483+0000", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9126999999999999e-05 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.923490+0000", 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1840000000000002e-06 2024-03-26T21:39:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.296612+0000", 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3731222380000001 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.296685+0000", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2589999999999994e-05 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.296756+0000", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1711999999999994e-05 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301057+0000", 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043003809999999998 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301149+0000", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1841000000000004e-05 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301693+0000", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000544365 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301734+0000", 2024-03-26T21:39:27.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1134999999999998e-05 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301801+0000", 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6400000000000001e-05 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197559 7.1 7:8b8d33fe:::benchmark_data_smithi055_81491_object78072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.944448+0000", 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.65906198099998, 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.357477166, 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197559 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944448+0000", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944448+0000", 2024-03-26T21:39:27.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944450+0000", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5600000000000001e-06 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944455+0000", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1889999999999998e-06 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944466+0000", 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1079e-05 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944474+0000", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4780000000000002e-06 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944552+0000", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8566000000000006e-05 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.944560+0000", 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0530000000000007e-06 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.298876+0000", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3543159300000001 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.298954+0000", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7705000000000002e-05 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.299037+0000", 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2911999999999995e-05 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301463+0000", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024262279999999999 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301663+0000", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020015300000000001 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301843+0000", 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017954000000000001 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301877+0000", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4363999999999998e-05 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301926+0000", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8408999999999997e-05 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197591 7.1 7:84a97563:::benchmark_data_smithi055_81491_object78104:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.971559+0000", 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.63195147800002, 2024-03-26T21:39:27.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3421253690000001, 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197591 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971559+0000", 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971559+0000", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971560+0000", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3549999999999999e-06 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971566+0000", 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9900000000000002e-06 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971576+0000", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0224000000000001e-05 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971583+0000", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8199999999999999e-06 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971634+0000", 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1116000000000003e-05 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.971643+0000", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9900000000000003e-06 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.301957+0000", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.330313743 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.302023+0000", 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6260000000000006e-05 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.302097+0000", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3328e-05 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.307188+0000", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050912450000000003 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.309726+0000", 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025384819999999999 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313529+0000", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038025419999999999 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313587+0000", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8489000000000002e-05 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313684+0000", 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6785000000000003e-05 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197605 7.1 7:8a6f74ab:::benchmark_data_smithi055_81491_object78118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:48.983376+0000", 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.62013399300002, 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3305171929999999, 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197605 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.656 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983376+0000", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983376+0000", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983378+0000", 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.19e-06 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983382+0000", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4039999999999997e-06 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983393+0000", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0732e-05 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983399+0000", 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2280000000000003e-06 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983453+0000", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4437000000000001e-05 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:48.983464+0000", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0530000000000001e-05 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.303221+0000", 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3197568959999999 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.303334+0000", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113039 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.303447+0000", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112741 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.309781+0000", 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063341389999999999 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313487+0000", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003706207 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313740+0000", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025266900000000002 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313796+0000", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6011999999999998e-05 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313894+0000", 2024-03-26T21:39:27.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7968999999999997e-05 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197636 7.1 7:8cfb2374:::benchmark_data_smithi055_81491_object78149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.011576+0000", 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.591934176, 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.30717525, 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197636 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011576+0000", 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011576+0000", 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011578+0000", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.604e-06 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011584+0000", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8719999999999999e-06 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011597+0000", 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2818e-05 2024-03-26T21:39:27.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011604+0000", 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5650000000000004e-06 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011706+0000", 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010231800000000001 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.011716+0000", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2639999999999997e-06 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.305207+0000", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.29349099 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.305318+0000", 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011150900000000001 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.305449+0000", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130906 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313909+0000", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084601869999999992 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313983+0000", 2024-03-26T21:39:27.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3781000000000004e-05 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318589+0000", 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004606382 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318636+0000", 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6777999999999998e-05 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318751+0000", 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115276 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197644 7.1 7:8d1a2689:::benchmark_data_smithi055_81491_object78157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.019856+0000", 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.583654325, 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992080269999999, 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197644 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019856+0000", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019856+0000", 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019857+0000", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1620000000000001e-06 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019862+0000", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019871+0000", 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4259999999999992e-06 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.019888+0000", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7132000000000001e-05 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.020100+0000", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211657 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.020114+0000", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3747000000000001e-05 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.307228+0000", 2024-03-26T21:39:27.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.287113663 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.307344+0000", 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116043 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.307449+0000", 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010562500000000001 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313916+0000", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064671759999999998 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.314059+0000", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014278500000000001 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318902+0000", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048432429999999997 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318970+0000", 2024-03-26T21:39:27.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7828000000000002e-05 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319064+0000", 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3888000000000006e-05 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197652 7.1 7:86a71c89:::benchmark_data_smithi055_81491_object78165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.027258+0000", 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.57625207500001, 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3074633090000001, 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197652 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027258+0000", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027258+0000", 2024-03-26T21:39:27.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027259+0000", 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1540000000000001e-06 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027264+0000", 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3100000000000002e-06 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027277+0000", 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3658e-05 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027283+0000", 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9449999999999998e-06 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027336+0000", 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2167999999999999e-05 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.027345+0000", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8570000000000007e-06 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.309819+0000", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2824731899999999 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.309926+0000", 2024-03-26T21:39:27.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107608 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.310001+0000", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4716999999999996e-05 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313924+0000", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039233360000000004 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318848+0000", 2024-03-26T21:39:27.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049241340000000001 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334656+0000", 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015807419 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334689+0000", 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3436000000000003e-05 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334722+0000", 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2376999999999998e-05 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197656 7.1 7:80280048:::benchmark_data_smithi055_81491_object78169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.031929+0000", 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.57158156200001, 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3029865410000001, 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.683 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197656 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031929+0000", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031929+0000", 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031930+0000", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4339999999999999e-06 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031937+0000", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0890000000000002e-06 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031948+0000", 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0569e-05 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.031955+0000", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5209999999999997e-06 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.032399+0000", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044347000000000001 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.032408+0000", 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9109999999999999e-06 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.311415+0000", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2790071590000001 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.311535+0000", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012034799999999999 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.311650+0000", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011483 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.313930+0000", 2024-03-26T21:39:27.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022801200000000001 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319123+0000", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051923500000000001 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334867+0000", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015744365999999999 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334892+0000", 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5049999999999999e-05 2024-03-26T21:39:27.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334915+0000", 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3323999999999999e-05 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197673 7.1 7:8b72d3d8:::benchmark_data_smithi055_81491_object78186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.048920+0000", 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.55459007500002, 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.286159008, 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197673 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048920+0000", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048920+0000", 2024-03-26T21:39:27.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048921+0000", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.156e-06 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048927+0000", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.524e-06 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048937+0000", 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0326000000000001e-05 2024-03-26T21:39:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048943+0000", 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.879e-06 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048986+0000", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2465000000000001e-05 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.048994+0000", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5210000000000002e-06 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.314106+0000", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2651118260000001 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.314226+0000", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011985900000000001 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.314340+0000", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114509 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.318866+0000", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045254359999999999 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319169+0000", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0003032 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.334964+0000", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015795035999999998 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335021+0000", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7380000000000003e-05 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335079+0000", 2024-03-26T21:39:27.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7890999999999999e-05 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197681 7.1 7:8603bc7b:::benchmark_data_smithi055_81491_object78194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.056256+0000", 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.547253958, 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.278905709, 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197681 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056256+0000", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056256+0000", 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056258+0000", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5629999999999999e-06 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056264+0000", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8780000000000003e-06 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056277+0000", 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3192e-05 2024-03-26T21:39:27.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056285+0000", 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1699999999999997e-06 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056333+0000", 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7525999999999999e-05 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.056339+0000", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7490000000000004e-06 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.316364+0000", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2600257830000001 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.316479+0000", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114602 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.316596+0000", 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116606 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319077+0000", 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024819769999999998 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319212+0000", 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000134981 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335115+0000", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015902218999999999 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335142+0000", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6832e-05 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335162+0000", 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0630999999999999e-05 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197687 7.1 7:83f3c96e:::benchmark_data_smithi055_81491_object78200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.064613+0000", 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.53889721399997, 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.270687699, 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.703 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197687 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064613+0000", 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064613+0000", 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064614+0000", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4099999999999997e-07 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064618+0000", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7469999999999999e-06 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064626+0000", 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0399999999999993e-06 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:27.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064631+0000", 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.271e-06 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064718+0000", 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6646999999999999e-05 2024-03-26T21:39:27.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.064727+0000", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4099999999999997e-06 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319269+0000", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2545414349999999 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319394+0000", 2024-03-26T21:39:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012569400000000001 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.319521+0000", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012619199999999999 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.323886+0000", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043654849999999997 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.324242+0000", 2024-03-26T21:39:27.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035567500000000001 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335206+0000", 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010964242000000001 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335258+0000", 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2533e-05 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335301+0000", 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2386999999999999e-05 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197721 7.1 7:8035715f:::benchmark_data_smithi055_81491_object78234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.128103+0000", 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.47540701999998, 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2073283509999999, 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197721 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128103+0000", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128103+0000", 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128105+0000", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128110+0000", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1050000000000003e-06 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128120+0000", 2024-03-26T21:39:27.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.076e-05 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128126+0000", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7849999999999997e-06 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128175+0000", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8637000000000002e-05 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.128186+0000", 2024-03-26T21:39:27.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0900000000000001e-05 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.320967+0000", 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1927807180000001 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.321102+0000", 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135029 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.321174+0000", 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2438e-05 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.324248+0000", 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030742650000000001 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.324365+0000", 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116836 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335352+0000", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010986553 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335391+0000", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9724000000000001e-05 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335432+0000", 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0351e-05 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.24704.0:197732 7.1 7:86111c85:::benchmark_data_smithi055_81491_object78245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:33:49.162140+0000", 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 338.44137000699999, 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173432757, 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.24704", 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 197732 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.718 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162140+0000", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162140+0000", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162141+0000", 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.08e-06 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162148+0000", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2709999999999998e-06 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162157+0000", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5389999999999996e-06 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162163+0000", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1109999999999996e-06 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162200+0000", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6579999999999999e-05 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:49.162207+0000", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.652e-06 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.322423+0000", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.160216484 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.322496+0000", 2024-03-26T21:39:27.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2892999999999995e-05 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.322562+0000", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6129000000000003e-05 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.324253+0000", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016905850000000001 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.324532+0000", 2024-03-26T21:39:27.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027966699999999999 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335481+0000", 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010948803 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335535+0000", 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3430000000000002e-05 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:39:27.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:33:50.335573+0000", 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8532999999999999e-05 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:39:27.724 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:39:27.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:27.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:27.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:27.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 6 16 17703 17687 184.224 162.312 0.0037282 0.00542044 2024-03-26T21:39:27.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:27.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:28.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:39:28.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:28.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:28.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:28.336 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:28.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:28.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:28.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:28.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:39:28.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:39:28.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:28.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:39:28.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:39:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:28 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:39:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:28 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:39:28.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.539 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:28.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:28.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:28.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:39:28.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:28.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:28.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:28.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:28.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 7 16 19922 19906 177.715 138.688 0.00383399 0.00562202 2024-03-26T21:39:29.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:29.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:29.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:29.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:29.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:29.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:29 smithi067 ceph-mon[33346]: pgmap v471: 148 pgs: 148 active+clean; 800 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T21:39:29.506 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:29 smithi055 ceph-mon[40587]: pgmap v471: 148 pgs: 148 active+clean; 800 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T21:39:29.506 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:29 smithi055 ceph-mon[29406]: pgmap v471: 148 pgs: 148 active+clean; 800 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T21:39:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:29.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:29.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:29.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:29.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:29.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:29.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 8 16 22224 22208 173.483 143.875 0.00716461 0.00575738 2024-03-26T21:39:30.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:30.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:30.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:30.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:30.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:30.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:30.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:39:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:39:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:39:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:39:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:39:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:30 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:39:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:30.514 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:39:30.514 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:39:30.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:39:30.516 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:39:30.516 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:39:30.516 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:39:30.516 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:30 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:39:30.599 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:30.605 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:30.653 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:30.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:30.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:30.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:30.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:30.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:30.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 9 15 24384 24369 169.213 135.062 0.00689618 0.00590416 2024-03-26T21:39:31.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:31.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:31.196 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:31.266 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:39:31.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:31.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:31 smithi055 ceph-mon[29406]: pgmap v472: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 992 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T21:39:31.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:31 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:31.314 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:31 smithi055 ceph-mon[40587]: pgmap v472: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 992 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T21:39:31.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:31 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:31.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:31.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:31.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:31.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:31.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:31 smithi067 ceph-mon[33346]: pgmap v472: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 992 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-26T21:39:31.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:31 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:31.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:31.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:31.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:31.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 10 16 26749 26733 167.066 147.75 0.00360009 0.00598044 2024-03-26T21:39:31.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:32.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:32.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:32.167 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:32.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:32.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:32.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:32.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:32.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:39:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:39:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: pgmap v474: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:39:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:39:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:39:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: pgmap v474: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T21:39:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:32.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:32.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:32.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:39:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:39:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2369223491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: pgmap v474: 148 pgs: 148 active+clean; 967 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:32.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:32.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:32.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:32.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:32.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 11 16 28751 28735 163.252 125.125 0.00694538 0.00611998 2024-03-26T21:39:32.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:32.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:33.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:33.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:33.273 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:33.374 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:33.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:33.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:33.535 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:39:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:39:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:33 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:39:33.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:33.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:33.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:33 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:39:33.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:33 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:39:33.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:33 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:39:33.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:33 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:39:33.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:33.802 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:33.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:33.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:33.985 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 12 15 30809 30794 160.37 128.688 0.0101491 0.00623037 2024-03-26T21:39:34.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:34.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:34.214 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:34.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:34.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:34.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:34.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:39:34.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:39:34.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 2.0 deep-scrub starts 2024-03-26T21:39:34.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 2.0 deep-scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[40587]: pgmap v475: 148 pgs: 148 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 2.0 deep-scrub starts 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 2.0 deep-scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:39:34.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:39:34.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:39:34.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:34 smithi055 ceph-mon[29406]: pgmap v475: 148 pgs: 148 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-26T21:39:34.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:34.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 2.0 deep-scrub starts 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 2.0 deep-scrub ok 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:39:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:39:34.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:39:34.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:39:34.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:34 smithi067 ceph-mon[33346]: pgmap v475: 148 pgs: 148 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.58k op/s 2024-03-26T21:39:34.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:34.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:34.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:34.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:34.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:34.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:34.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 13 16 32305 32289 155.209 93.4375 0.00426075 0.0064366 2024-03-26T21:39:35.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:35.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:35.137 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:35.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:35.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:35.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:35.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:35.539 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:35.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:35.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:35.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[29406]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:39:35.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:39:35.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:39:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[40587]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:39:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:39:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:35 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:39:35.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:35.680 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:35 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:35 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:35 smithi067 ceph-mon[33346]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T21:39:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:35 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:39:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:35 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:39:35.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:35.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:35.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:35.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:35.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:35.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 14 16 33692 33676 150.314 86.6875 0.0041277 0.00663842 2024-03-26T21:39:36.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:36.083 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:36.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:36.222 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:36.324 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:36.347 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:36.449 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:36.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:36.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:36.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:36.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:36 smithi067 ceph-mon[33346]: pgmap v477: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:39:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:36 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:39:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:36 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:39:36.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:36.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[40587]: pgmap v477: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:39:36.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:39:36.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:39:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[29406]: pgmap v477: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:39:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:39:36.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:36 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:39:36.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:36.890 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:36.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:36.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 15 16 35135 35119 146.306 90.1875 0.00884657 0.00682922 2024-03-26T21:39:36.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:37.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:37.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:37.171 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:37.273 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:37.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:37.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:37.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:37.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:37.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:37.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:37.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:39:37.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:37.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:39:37.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T21:39:37.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:37.839 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:37.941 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:37.967 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:37.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:37.986 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 16 16 36731 36715 143.396 99.75 0.00604239 0.0069685 2024-03-26T21:39:38.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:38.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:38.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:38.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:38.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:38.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:38 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:39:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:38 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:39:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:38 smithi067 ceph-mon[33346]: pgmap v479: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-26T21:39:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:38 smithi067 ceph-mon[33346]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:39:38.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:39:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:39:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[29406]: pgmap v479: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-26T21:39:38.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[29406]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:39:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:39:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:39:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[40587]: pgmap v479: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 185 MiB/s wr, 2.96k op/s 2024-03-26T21:39:38.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-mon[40587]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T21:39:38.834 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:38.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:38.976 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:38.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:38.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 17 16 38602 38586 141.839 116.938 0.0056645 0.00704464 2024-03-26T21:39:39.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:39.183 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:39.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:39.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:39.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:39.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:39:39.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:39 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:39:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:39 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:39:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:39 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:39:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:39 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:39:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:39 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:39:39.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:39.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:39.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:39.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:39.852 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:39.880 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:39:39.925 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:39:39.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:39.963 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:39:39.986 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:39.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 18 16 40155 40139 139.351 97.0625 0.00947467 0.00716891 2024-03-26T21:39:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:39.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:40.004 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:39:40.045 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:39:40.087 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:39:40.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:40.129 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:39:40.171 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:39:40.175 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:39:40.176 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:39:40.176 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:39:40.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:40.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:40.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:40.406 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:40.508 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.553 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:40.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:40.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:40.671 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 5.18 deep-scrub starts 2024-03-26T21:39:40.671 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 5.18 deep-scrub ok 2024-03-26T21:39:40.671 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:39:40.672 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:39:40.672 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: pgmap v481: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-26T21:39:40.672 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/658760899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:40.672 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:39:40.672 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:40 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:39:40.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:40.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:40.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 5.18 deep-scrub starts 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 5.18 deep-scrub ok 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: pgmap v481: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/658760899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 5.18 deep-scrub starts 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 5.18 deep-scrub ok 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:39:40.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: pgmap v481: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-26T21:39:40.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/658760899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:40.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:39:40.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:40 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:39:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:40.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:40.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:39:40.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:40.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:40.987 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 19 16 41800 41784 137.423 102.812 0.0121943 0.00727136 2024-03-26T21:39:41.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:41.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:41.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:41.372 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:41.419 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:41.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:41.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:41.658 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:41.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:41.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:41.912 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:41.912 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:41.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:41.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:41.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:39:41.996054+0000 min lat: 0.00115728 max lat: 0.188069 avg lat: 0.00736141 2024-03-26T21:39:41.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:39:41.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 20 16 43462 43446 135.745 103.875 0.0081182 0.00736141 2024-03-26T21:39:42.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:42.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:42.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:42.237 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:42.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:42.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:42.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:42.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:42.619 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:42.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:42.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:42.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:42 smithi055 ceph-mon[40587]: pgmap v482: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 710 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T21:39:42.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:42 smithi055 ceph-mon[29406]: pgmap v482: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 710 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T21:39:42.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:42.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:42.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:42.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 21 15 45329 45314 134.84 116.75 0.00858473 0.00741145 2024-03-26T21:39:42.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:42 smithi067 ceph-mon[33346]: pgmap v482: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 710 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T21:39:43.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:43.070 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:39:43.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:43.132 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:43.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:43.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.353 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:43.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:43.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:43.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:43.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:43.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:43.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:43.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:39:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:39:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:39:43.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:39:43.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:43 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:39:43.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:43.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:43.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 22 16 47054 47038 133.608 107.75 0.00858028 0.00747955 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:39:43.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:43 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:39:44.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:44.063 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:44.098 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:44.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:44.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.341 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:44.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:44.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:39:44.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:39:44.476 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:39:44.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:44.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:44.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:44.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[29406]: 5.1c deep-scrub starts 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[29406]: 5.1c deep-scrub ok 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[29406]: pgmap v483: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[40587]: 5.1c deep-scrub starts 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[40587]: 5.1c deep-scrub ok 2024-03-26T21:39:44.772 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:44 smithi055 ceph-mon[40587]: pgmap v483: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T21:39:44.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:44.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:44.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:44.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 23 16 48755 48739 132.421 106.312 0.00772572 0.00754657 2024-03-26T21:39:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:44 smithi067 ceph-mon[33346]: 5.1c deep-scrub starts 2024-03-26T21:39:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:44 smithi067 ceph-mon[33346]: 5.1c deep-scrub ok 2024-03-26T21:39:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:44 smithi067 ceph-mon[33346]: pgmap v483: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-26T21:39:45.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:45.051 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:45.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:45.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:45.332 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:45.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:45.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:45.525 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:39:45.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:39:45.525 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:39:45.525 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T21:39:45.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7 2024-03-26T21:39:45.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:45.610 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:45.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.750 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:45.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:45.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:39:45.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:39:45.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:39:45.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:39:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:39:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:39:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:39:45.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:45 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:39:45.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:45.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:45.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:45 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:39:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:45 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:39:45.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:45 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:39:45.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:45 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:39:45.990 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:45 smithi067 systemd[1]: Stopping Ceph osd.7 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:39:45.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 24 15 50415 50400 131.229 103.812 0.0322427 0.00761576 2024-03-26T21:39:45.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:46.060 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:46.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:46.201 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:46.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:46.336 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:46.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:46.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[55899]: 2024-03-26T21:39:46.115+0000 7f477dc37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:39:46.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[55899]: 2024-03-26T21:39:46.115+0000 7f477dc37700 -1 osd.7 248 *** Got signal Terminated *** 2024-03-26T21:39:46.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[55899]: 2024-03-26T21:39:46.115+0000 7f477dc37700 -1 osd.7 248 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:39:46.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:46.606 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:39:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:46.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:46.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.734 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:46.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:39:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:39:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:39:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: pgmap v484: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: osd.7 marked itself down and dead 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: pgmap v484: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: osd.7 marked itself down and dead 2024-03-26T21:39:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:46 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:39:46.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:46.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:46.863 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:46.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:46.984 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:46.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:46.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 25 16 51386 51370 128.404 60.625 0.00273755 0.00766611 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: pgmap v484: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: osd.7 marked itself down and dead 2024-03-26T21:39:46.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:46 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:39:47.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:47.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:47.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:47.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:47.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:47.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:47.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:47.466 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:47 smithi067 podman[109620]: 2024-03-26 21:39:47.182063573 +0000 UTC m=+1.330076086 container died e0a00657d9514aaae8b06ad5ef5dd5fdeffed9bbce69f15dc516e929d2cd2d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:39:47.485 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:47.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:47.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:47.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:47.739 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:47 smithi067 podman[109620]: 2024-03-26 21:39:47.564867404 +0000 UTC m=+1.712879921 container cleanup e0a00657d9514aaae8b06ad5ef5dd5fdeffed9bbce69f15dc516e929d2cd2d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:39:47.739 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:47 smithi067 bash[109620]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:47.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:47 smithi067 ceph-mon[33346]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T21:39:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:47.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:47.832 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[40587]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T21:39:47.833 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2091060325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:47.834 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:47 smithi055 ceph-mon[29406]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T21:39:47.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:47.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:47.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:47.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 26 16 51888 51872 124.673 31.375 0.0189687 0.00801558 2024-03-26T21:39:47.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:48.037 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:48.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:48.178 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:48.220 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:39:48.220 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:39:48.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.311 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:48.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:48.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:48.438 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:48.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:48.584 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:48.670 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:48 smithi055 ceph-mon[40587]: pgmap v486: 148 pgs: 31 peering, 12 stale+active+clean, 105 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-03-26T21:39:48.670 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:48 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:39:48.671 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:48 smithi055 ceph-mon[29406]: pgmap v486: 148 pgs: 31 peering, 12 stale+active+clean, 105 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-03-26T21:39:48.671 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:48 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:39:48.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:48.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:48.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:48.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:48.984 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:48 smithi067 ceph-mon[33346]: pgmap v486: 148 pgs: 31 peering, 12 stale+active+clean, 105 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-03-26T21:39:48.984 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:48 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:39:48.984 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:48 smithi067 podman[109721]: 2024-03-26 21:39:48.583796775 +0000 UTC m=+1.398833370 container remove e0a00657d9514aaae8b06ad5ef5dd5fdeffed9bbce69f15dc516e929d2cd2d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T21:39:48.988 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 27 16 53525 53509 123.844 102.312 0.0162488 0.0080698 2024-03-26T21:39:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:48.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:49.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:49.137 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:49.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:49.240 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:48 smithi067 podman[109892]: 2024-03-26 21:39:48.896991905 +0000 UTC m=+0.027251188 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:39:49.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:49.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:39:49.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:49.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:39:49.503 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:49.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:49.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:49.673 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:49 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 9796/155529 objects degraded (6.299%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.685 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:49.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:49.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:49.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:49.949 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:49.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:49.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:49.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:49 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 9796/155529 objects degraded (6.299%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:49.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 28 16 54829 54813 122.329 81.5 0.0177076 0.00816829 2024-03-26T21:39:50.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:49 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 9796/155529 objects degraded (6.299%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:50.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:50.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:50.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:50.283 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:50.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:50.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:50.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:50 smithi067 podman[109892]: 2024-03-26 21:39:50.110507803 +0000 UTC m=+1.240767064 container create ffec6913cfc557a5d48952de08f66dbd59c05aa93355c63541cdd01640105e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-26T21:39:50.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:50.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:50.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:50.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:50.898 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:50 smithi067 ceph-mon[33346]: pgmap v488: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:50.898 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:50 smithi067 podman[109892]: 2024-03-26 21:39:50.780884177 +0000 UTC m=+1.911143439 container init ffec6913cfc557a5d48952de08f66dbd59c05aa93355c63541cdd01640105e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-26T21:39:50.898 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:50 smithi067 podman[109892]: 2024-03-26 21:39:50.785450016 +0000 UTC m=+1.915709286 container start ffec6913cfc557a5d48952de08f66dbd59c05aa93355c63541cdd01640105e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:39:50.918 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:50 smithi055 ceph-mon[40587]: pgmap v488: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:50.918 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:50 smithi055 ceph-mon[29406]: pgmap v488: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:50.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:50.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:50.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 29 16 56344 56328 121.376 94.6875 0.00976753 0.00823409 2024-03-26T21:39:51.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:51.098 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:51.184 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:50 smithi067 podman[109892]: 2024-03-26 21:39:50.905918034 +0000 UTC m=+2.036177299 container attach ffec6913cfc557a5d48952de08f66dbd59c05aa93355c63541cdd01640105e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T21:39:51.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:51.231 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:39:51.245 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:39:51.246 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:39:51.246 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:39:51.246 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:39:51.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:51.372 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:51.457 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:39:51 smithi067 podman[109892]: 2024-03-26 21:39:51.191393043 +0000 UTC m=+2.321652307 container died ffec6913cfc557a5d48952de08f66dbd59c05aa93355c63541cdd01640105e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T21:39:51.473 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:51.516 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:51.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:51.647 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4087765465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:51.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:51.793 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:51.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4087765465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:51.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4087765465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:39:51.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:51.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:51.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:51.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T21:39:51.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 30 16 57944 57928 120.663 100 0.00713422 0.00828286 2024-03-26T21:39:52.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:39:52.024 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T21:39:52.024 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-03-26T21:39:52.024 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi067.front.sepia.ceph.com 2024-03-26T21:39:52.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.7 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-03-26T21:39:52.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:52.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:52.287 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T21:39:52.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:52.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:52.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:52.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:52.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:52.917 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:52.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:52.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:52 smithi067 ceph-mon[33346]: pgmap v489: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:52.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 31 15 59575 59560 120.061 102 0.0146561 0.00832429 2024-03-26T21:39:53.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:53.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:52 smithi055 ceph-mon[40587]: pgmap v489: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:53.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:52 smithi055 ceph-mon[29406]: pgmap v489: 148 pgs: 10 active+undersized+degraded, 31 peering, 4 stale+active+clean, 12 active+undersized, 91 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 9796/155529 objects degraded (6.299%); 7.8 KiB/s, 0 objects/s recovering 2024-03-26T21:39:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:53.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:53.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:53.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:53.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:53.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:53.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:39:53.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:53.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:53 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:53.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:53.812 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:53.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:53 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:53 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:39:53.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:53.944 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:53.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:53.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 32 16 60875 60859 118.846 81.1875 0.00937132 0.00840961 2024-03-26T21:39:54.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:54.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:54.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:54.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:54.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:54.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:54.475 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:54.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:54.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:54.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:54.760 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:54.862 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:54.902 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:54.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s; 32498/175011 objects degraded (18.569%); 24 KiB/s, 0 objects/s recovering 2024-03-26T21:39:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-03-26T21:39:54.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi067 to 3248M 2024-03-26T21:39:54.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:54 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:54.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 33 16 62224 62208 117.799 84.3125 0.0119199 0.00848402 2024-03-26T21:39:55.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s; 32498/175011 objects degraded (18.569%); 24 KiB/s, 0 objects/s recovering 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi067 to 3248M 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s; 32498/175011 objects degraded (18.569%); 24 KiB/s, 0 objects/s recovering 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi067 to 3248M 2024-03-26T21:39:55.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:54 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:55.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:55.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:39:55.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:39:55.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.330 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:55.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:55.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:55.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:55.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:55.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:55.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.761 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:55.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:55.862 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:55.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:55.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:55.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:55 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:55.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 34 16 63649 63633 116.954 89.0625 0.0149871 0.00854546 2024-03-26T21:39:55.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:56.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:56.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:56.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:39:56.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:55 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:39:56.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:39:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:39:56.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:56.318 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:56.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:56.460 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:56.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:56.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:56.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:56.732 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:56.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:56.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:56.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 35 16 65072 65056 116.153 88.9375 0.0162835 0.00860383 2024-03-26T21:39:56.995 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:56.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:57.060 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:56 smithi067 ceph-mon[33346]: pgmap v491: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 32983/176466 objects degraded (18.691%); 21 KiB/s, 0 objects/s recovering 2024-03-26T21:39:57.060 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:56 smithi067 ceph-mon[33346]: Adjusting osd_memory_target on smithi055 to 3249M 2024-03-26T21:39:57.060 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:56 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 32983/176466 objects degraded (18.691%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:57.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[29406]: pgmap v491: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 32983/176466 objects degraded (18.691%); 21 KiB/s, 0 objects/s recovering 2024-03-26T21:39:57.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[29406]: Adjusting osd_memory_target on smithi055 to 3249M 2024-03-26T21:39:57.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 32983/176466 objects degraded (18.691%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:57.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[40587]: pgmap v491: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 32983/176466 objects degraded (18.691%); 21 KiB/s, 0 objects/s recovering 2024-03-26T21:39:57.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[40587]: Adjusting osd_memory_target on smithi055 to 3249M 2024-03-26T21:39:57.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:56 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 32983/176466 objects degraded (18.691%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:39:57.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:57.134 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:57.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:39:57.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:57.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:57.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:57.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:57.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:39:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:39:57.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:57.541 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:57.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:57.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:57.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:57.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:57.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:57.981 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:57.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:57.989 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 36 16 66289 66273 115.039 76.0625 0.00417524 0.0086873 2024-03-26T21:39:58.036 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:58.036 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:58.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:58.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:58.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:39:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.259 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:58.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:58.305 INFO:teuthology.orchestra.run.smithi067.stdout:9.3 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.b 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.9 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.6 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.3 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:9.a 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.1c 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.18 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.16 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.12 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:5.9 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:5.6 2024-03-26T21:39:58.306 INFO:teuthology.orchestra.run.smithi067.stdout:6.14 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:5.4 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:4.0 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:6.15 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:4.a 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.f 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:4.8 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.9 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:4.15 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.11 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.8 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:6.1b 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.7 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:6.5 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:3.e 2024-03-26T21:39:58.307 INFO:teuthology.orchestra.run.smithi067.stdout:4.16 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:4.1e 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:5.10 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:5.11 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.16 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.1b 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.15 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.13 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:4.5 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:5.1f 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:1.0 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:2.0 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.d 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:3.12 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:9.e 2024-03-26T21:39:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:4.3 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:4.11 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.19 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.5 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.d 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:4.17 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:3.14 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:4.14 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.1e 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.14 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.d 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.7 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.17 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.1 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.9 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.1 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:5.1b 2024-03-26T21:39:58.309 INFO:teuthology.orchestra.run.smithi067.stdout:9.c 2024-03-26T21:39:58.310 INFO:teuthology.orchestra.run.smithi067.stdout:6.8 2024-03-26T21:39:58.310 INFO:teuthology.orchestra.run.smithi067.stdout:5.1a 2024-03-26T21:39:58.360 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:39:58.378 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:39:58.401 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:39:58.401 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:39:58.401 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:39:58.401 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197736 7.9 7:974c4e52:::benchmark_data_smithi055_81491_object78249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.10056494999998, 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.65874689099999995, 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197736 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.402 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302064+0000", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302065+0000", 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3850000000000001e-06 2024-03-26T21:39:58.403 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.405 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302072+0000", 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2169999999999997e-06 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302085+0000", 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2845e-05 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.406 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302095+0000", 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0302999999999999e-05 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302199+0000", 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103849 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.302211+0000", 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1514e-05 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954115+0000", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.65190433400000003 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954255+0000", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000139825 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.954396+0000", 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000141545 2024-03-26T21:39:58.408 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956845+0000", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024481049999999999 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958788+0000", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001942924 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960742+0000", 2024-03-26T21:39:58.409 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0019545600000000002 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960775+0000", 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2493000000000002e-05 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960811+0000", 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5991999999999997e-05 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.410 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197747 7.9 7:9a8570b5:::benchmark_data_smithi055_81491_object78260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.08329041299999, 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.64169457200000002, 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197747 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.411 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319338+0000", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319340+0000", 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1229999999999998e-06 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319361+0000", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0188000000000001e-05 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319432+0000", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1435999999999998e-05 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319444+0000", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1871999999999999e-05 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319586+0000", 2024-03-26T21:39:58.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000142278 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.319597+0000", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1341e-05 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956876+0000", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63727853700000003 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.956953+0000", 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7040999999999996e-05 2024-03-26T21:39:58.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.957055+0000", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010241700000000001 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960899+0000", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038431479999999998 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960953+0000", 2024-03-26T21:39:58.415 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4721e-05 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960962+0000", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7830000000000004e-06 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.960990+0000", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7970000000000002e-05 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961033+0000", 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2716999999999998e-05 2024-03-26T21:39:58.416 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197755 7.9 7:9cc27710:::benchmark_data_smithi055_81491_object78268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.07301558099999, 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63164425599999996, 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.417 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197755 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329613+0000", 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.418 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329614+0000", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6299999999999993e-07 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329620+0000", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7230000000000001e-06 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329630+0000", 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0569e-05 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329638+0000", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8010000000000004e-06 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329725+0000", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6638999999999995e-05 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.329734+0000", 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2809999999999997e-06 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.420 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958820+0000", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62908593499999998 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958885+0000", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5277000000000005e-05 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.958951+0000", 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5949000000000002e-05 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961041+0000", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020901750000000001 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961088+0000", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6925999999999998e-05 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961168+0000", 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9276000000000005e-05 2024-03-26T21:39:58.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961219+0000", 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1544e-05 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961257+0000", 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8198000000000003e-05 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197762 7.9 7:9b3826cf:::benchmark_data_smithi055_81491_object78275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.06713060300001, 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63713577799999999, 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197762 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.424 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335498+0000", 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335499+0000", 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4839999999999999e-06 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.425 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335505+0000", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3909999999999997e-06 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335515+0000", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0103e-05 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335523+0000", 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1610000000000003e-06 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335590+0000", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6483999999999996e-05 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335598+0000", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6160000000000002e-06 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961364+0000", 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62576607200000001 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961461+0000", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6230000000000005e-05 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.961528+0000", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7299999999999996e-05 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972488+0000", 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.01096025 2024-03-26T21:39:58.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972501+0000", 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.307e-05 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972538+0000", 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6834e-05 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972589+0000", 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1484999999999999e-05 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972634+0000", 2024-03-26T21:39:58.430 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4298000000000002e-05 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197765 7.9 7:9de1617a:::benchmark_data_smithi055_81491_object78278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.067042356, 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.63724420299999995, 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.431 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197765 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335586+0000", 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.432 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335588+0000", 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4139999999999999e-06 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335594+0000", 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4960000000000001e-06 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335607+0000", 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2513e-05 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335615+0000", 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8070000000000006e-06 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335683+0000", 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7102999999999998e-05 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.335691+0000", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5620000000000004e-06 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963592+0000", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62790077300000002 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963694+0000", 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010242399999999999 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.963804+0000", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109621 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972506+0000", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087021449999999997 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972674+0000", 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016797000000000001 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972747+0000", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3324999999999993e-05 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972795+0000", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7966000000000001e-05 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972830+0000", 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5083999999999998e-05 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.437 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197779 7.9 7:9aaf6e48:::benchmark_data_smithi055_81491_object78292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.048680864, 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62208113700000001, 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197779 2024-03-26T21:39:58.438 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353948+0000", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353949+0000", 2024-03-26T21:39:58.439 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5600000000000001e-06 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353956+0000", 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1930000000000004e-06 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353971+0000", 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4969e-05 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.353981+0000", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2650000000000002e-06 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.354171+0000", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019027399999999999 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.354181+0000", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9229999999999997e-06 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.441 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966463+0000", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.61228198 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966603+0000", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013977299999999999 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.966724+0000", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121178 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.442 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972643+0000", 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.005919059 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972714+0000", 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0844000000000001e-05 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.975915+0000", 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032011370000000002 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.975977+0000", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1993000000000001e-05 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976029+0000", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1989e-05 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197785 7.9 7:97132301:::benchmark_data_smithi055_81491_object78298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.04532504500003, 2024-03-26T21:39:58.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62510192499999995, 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197785 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357304+0000", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357305+0000", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.657e-06 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357312+0000", 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5239999999999997e-06 2024-03-26T21:39:58.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357328+0000", 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6161e-05 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357338+0000", 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6849999999999996e-06 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357447+0000", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109069 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.357457+0000", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0193e-05 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969600+0000", 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.612143192 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969722+0000", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121581 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.969844+0000", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000122569 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972685+0000", 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002840609 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972864+0000", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017964700000000001 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982308+0000", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0094434640000000004 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982361+0000", 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2800000000000003e-05 2024-03-26T21:39:58.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982405+0000", 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4774000000000002e-05 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197787 7.9 7:92feca94:::benchmark_data_smithi055_81491_object78300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.043556319, 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.62348025900000004, 2024-03-26T21:39:58.451 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197787 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359072+0000", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359073+0000", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.094e-06 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359079+0000", 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1209999999999998e-06 2024-03-26T21:39:58.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359090+0000", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1576000000000001e-05 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359098+0000", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1599999999999998e-06 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359158+0000", 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9351999999999997e-05 2024-03-26T21:39:58.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.359166+0000", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9670000000000001e-06 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972892+0000", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.61372603000000003 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.972997+0000", 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105892 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.973118+0000", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012077400000000001 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982176+0000", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0090576570000000002 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982219+0000", 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3080000000000001e-05 2024-03-26T21:39:58.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982465+0000", 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00024610700000000002 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982514+0000", 2024-03-26T21:39:58.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9280999999999998e-05 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982553+0000", 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8167999999999998e-05 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197819 7.9 7:958a9edc:::benchmark_data_smithi055_81491_object78332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:39:58.458 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.01238348499999, 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.59241795100000005, 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197819 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:39:58.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390245+0000", 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390247+0000", 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.401e-06 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390253+0000", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2020000000000001e-06 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390266+0000", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2906e-05 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390272+0000", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1279999999999996e-06 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390325+0000", 2024-03-26T21:39:58.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2904e-05 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.390335+0000", 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0115000000000001e-05 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976112+0000", 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.58577727000000002 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976265+0000", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000153177 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.976371+0000", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105565 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982232+0000", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058610119999999996 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982268+0000", 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5797999999999998e-05 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982592+0000", 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032408799999999999 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982631+0000", 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9313000000000001e-05 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.982663+0000", 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2072000000000001e-05 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197831 7.9 7:9fc20ae5:::benchmark_data_smithi055_81491_object78344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 368.00240243899998, 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.59832579299999999, 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.465 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197831 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.466 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400226+0000", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400227+0000", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0500000000000002e-07 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400231+0000", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0069999999999999e-06 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400241+0000", 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0274e-05 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400249+0000", 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8520000000000004e-06 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400413+0000", 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000163886 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.400421+0000", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9389999999999997e-06 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979179+0000", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.57875759599999999 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979291+0000", 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112336 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.979406+0000", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011535300000000001 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989307+0000", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0099010869999999994 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992452+0000", 2024-03-26T21:39:58.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0031449859999999998 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998396+0000", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0059438269999999996 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998492+0000", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6281999999999997e-05 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998552+0000", 2024-03-26T21:39:58.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9463000000000003e-05 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197855 7.9 7:9ae59bd5:::benchmark_data_smithi055_81491_object78368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.97190643599998, 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.57252650800000005, 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.472 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197855 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430722+0000", 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430723+0000", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7400000000000002e-07 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430728+0000", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9740000000000001e-06 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430736+0000", 2024-03-26T21:39:58.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0020000000000006e-06 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430743+0000", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7780000000000002e-06 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430814+0000", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0998e-05 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.430822+0000", 2024-03-26T21:39:58.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5890000000000003e-06 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.985967+0000", 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.55514456499999998 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.986084+0000", 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117447 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.986200+0000", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011593400000000001 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992507+0000", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0063063190000000003 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992520+0000", 2024-03-26T21:39:58.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.346e-05 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003156+0000", 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010636174 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003202+0000", 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6131999999999998e-05 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003249+0000", 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6261999999999999e-05 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197875 7.9 7:90cfef7a:::benchmark_data_smithi055_81491_object78388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.94983855700002, 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.55058782699999997, 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.479 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197875 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452790+0000", 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452792+0000", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.485e-06 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452797+0000", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9699999999999998e-06 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452810+0000", 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3777999999999999e-05 2024-03-26T21:39:58.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452820+0000", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4769999999999992e-06 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452882+0000", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1804000000000007e-05 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.452891+0000", 2024-03-26T21:39:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.098e-06 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989405+0000", 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.536514355 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989510+0000", 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104522 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.989625+0000", 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115179 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.484 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992561+0000", 2024-03-26T21:39:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.485 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:58.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:39:58.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029358219999999998 2024-03-26T21:39:58.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998566+0000", 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0060056479999999997 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003296+0000", 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0047299539999999998 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.487 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003340+0000", 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3852e-05 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003378+0000", 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7883000000000003e-05 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197878 7.9 7:990f5c53:::benchmark_data_smithi055_81491_object78391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.488 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.94832531999998, 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.54920376000000004, 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197878 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454303+0000", 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454304+0000", 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.048e-06 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454309+0000", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9910000000000002e-06 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454320+0000", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1080000000000001e-05 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454328+0000", 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6960000000000005e-06 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454387+0000", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8471000000000003e-05 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.454396+0000", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0589999999999992e-06 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992599+0000", 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53820327800000001 2024-03-26T21:39:58.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992702+0000", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103361 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.992811+0000", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000108515 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998620+0000", 2024-03-26T21:39:58.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058094109999999996 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998674+0000", 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3369999999999999e-05 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003426+0000", 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004752399 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003468+0000", 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2292000000000001e-05 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003507+0000", 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8788999999999997e-05 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197883 7.9 7:9dafc3e4:::benchmark_data_smithi055_81491_object78396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.94124642200001, 2024-03-26T21:39:58.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.54529854600000005, 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197883 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461382+0000", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461383+0000", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4899999999999995e-07 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461386+0000", 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3869999999999999e-06 2024-03-26T21:39:58.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461394+0000", 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3660000000000003e-06 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461400+0000", 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4749999999999998e-06 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461457+0000", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7148000000000002e-05 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.461466+0000", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2320000000000007e-06 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995439+0000", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53397292699999999 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995539+0000", 2024-03-26T21:39:58.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9877000000000006e-05 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.995657+0000", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117469 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998662+0000", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030051320000000002 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003259+0000", 2024-03-26T21:39:58.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004597584 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006449+0000", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003189103 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006631+0000", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018235899999999999 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006681+0000", 2024-03-26T21:39:58.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9838e-05 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197887 7.9 7:976766be:::benchmark_data_smithi055_81491_object78400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.93507867099999, 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53942058100000001, 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.502 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197887 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467550+0000", 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467551+0000", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3399999999999999e-07 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467554+0000", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4470000000000002e-06 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467562+0000", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.909e-06 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467567+0000", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9840000000000001e-06 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467609+0000", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1995e-05 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.467616+0000", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7499999999999997e-06 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998701+0000", 2024-03-26T21:39:58.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53108542199999997 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.998812+0000", 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000111204 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.000049+0000", 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0012364100000000001 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003387+0000", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033380749999999998 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006723+0000", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033363199999999998 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006761+0000", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8281999999999997e-05 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.507 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006926+0000", 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016495 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.006971+0000", 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4199000000000002e-05 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.24704.0:197891 7.9 7:92702fe6:::benchmark_data_smithi055_81491_object78404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:39:58.508 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 367.92440222300002, 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.53345969100000001, 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.24704", 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2492443911", 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 197891 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478226+0000", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478227+0000", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7800000000000001e-07 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478231+0000", 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9500000000000003e-06 2024-03-26T21:39:58.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478241+0000", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8020000000000003e-06 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478247+0000", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0469999999999999e-06 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478287+0000", 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0296000000000002e-05 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:50.478292+0000", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2070000000000003e-06 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003550+0000", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.52525799399999995 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003672+0000", 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121983 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.003796+0000", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012325500000000001 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.009910+0000", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0061143020000000003 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011217+0000", 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001307374 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011594+0000", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000376277 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011644+0000", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9979999999999999e-05 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:33:51.011686+0000", 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2546000000000002e-05 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.514 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:86638 8.e 8:79416a1d:::benchmark_data_smithi055_110126_object86637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:06.029531+0000", 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 232.37309724799999, 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.24330035699999999, 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 86638 2024-03-26T21:39:58.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029531+0000", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029531+0000", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029533+0000", 2024-03-26T21:39:58.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.35e-06 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029554+0000", 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1095e-05 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029564+0000", 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0013999999999999e-05 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.029571+0000", 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2409999999999998e-06 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183267+0000", 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15369558899999999 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183602+0000", 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00033525500000000001 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183803+0000", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000201008 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267735+0000", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.083931650999999996 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267750+0000", 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5916000000000001e-05 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267945+0000", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019467099999999999 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267996+0000", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0739000000000003e-05 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.272832+0000", 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0048358280000000003 2024-03-26T21:39:58.520 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:86643 8.6 8:695f9c9f:::benchmark_data_smithi055_110126_object86642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:06.032928+0000", 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 232.36970026500001, 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.234917607, 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:58.521 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 86643 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032928+0000", 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032928+0000", 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032930+0000", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.77e-06 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032958+0000", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7401000000000001e-05 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032971+0000", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3023000000000001e-05 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032980+0000", 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3009999999999995e-06 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183238+0000", 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15025814900000001 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183358+0000", 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011985300000000001 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 6,7", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.183502+0000", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014399500000000001 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.258560+0000", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.075057923999999998 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.258607+0000", 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6990999999999998e-05 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267758+0000", 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0091515799999999994 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267806+0000", 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7905999999999998e-05 2024-03-26T21:39:58.526 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.267846+0000", 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9714e-05 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.15096.0:86639 8.5 e191/188 8:a036fec7:::benchmark_data_smithi055_110126_object86638:head v 191'5366, mlcod=177'107)", 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:06.037447+0000", 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 232.36518168800001, 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.224081906, 2024-03-26T21:39:58.527 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037447+0000", 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037447+0000", 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037448+0000", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7100000000000001e-07 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037495+0000", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7077999999999999e-05 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037503+0000", 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6349999999999995e-06 2024-03-26T21:39:58.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.037510+0000", 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7349999999999999e-06 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.039006+0000", 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001495814 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.172256+0000", 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13325024399999999 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.261497+0000", 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.089240691999999996 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.261529+0000", 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2036999999999998e-05 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:108541 8.e 8:718666ab:::benchmark_data_smithi055_110126_object108540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:31.259192+0000", 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 207.14343677799999, 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.221581849, 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 108541 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.532 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259192+0000", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259192+0000", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259193+0000", 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2839999999999999e-06 2024-03-26T21:39:58.533 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259215+0000", 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2118999999999999e-05 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259226+0000", 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0882e-05 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:39:58.534 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.259234+0000", 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1990000000000006e-06 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.338093+0000", 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.078859156999999999 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.338263+0000", 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016979500000000001 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.535 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.338558+0000", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029506200000000002 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.346464+0000", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0079054910000000006 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.346558+0000", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4487999999999994e-05 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:39:58.536 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.476870+0000", 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13031163100000001 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.476948+0000", 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8066999999999996e-05 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:31.480774+0000", 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038256739999999998 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:39:58.537 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:39:58.538 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:39:58.587 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:39:58.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:39:58.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:58.774 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:58.798 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:39:58.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:58.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:58.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[29406]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 35561/192273 objects degraded (18.495%); 19 KiB/s, 0 objects/s recovering 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1379061422' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[40587]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 35561/192273 objects degraded (18.495%); 19 KiB/s, 0 objects/s recovering 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1379061422' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:58.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:58.991 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 37 15 67516 67501 114.004 76.75 0.00429132 0.0087673 2024-03-26T21:39:59.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:59.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:39:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:39:59.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:58 smithi067 ceph-mon[33346]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 35561/192273 objects degraded (18.495%); 19 KiB/s, 0 objects/s recovering 2024-03-26T21:39:59.223 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1379061422' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:59.223 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:39:59.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:59.298 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:39:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:39:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:39:59.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:59.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:39:59.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:39:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:59.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:39:59.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.7 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 4.14 --file /var/log/ceph/exp.4.14.7 2024-03-26T21:39:59.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:59.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:39:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T21:39:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:39:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:39:59.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:39:59.970 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:59 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:39:59.970 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:59 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:39:59.970 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:39:59 smithi067 ceph-mon[33346]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T21:39:59.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 38 16 68917 68901 113.307 87.5 0.0095725 0.00882098 2024-03-26T21:39:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:39:59.993 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:39:59.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:00.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:00.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:40:00.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[29406]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T21:40:00.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:00.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:40:00.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:39:59 smithi055 ceph-mon[40587]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T21:40:00.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:00.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:00.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:00.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:00.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:00.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:00.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:00.522 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:00.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:00.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:00.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:00.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:00.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:00.923 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:00.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:00.990 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 39 15 70133 70118 112.352 76.0625 0.0115798 0.00889649 2024-03-26T21:40:01.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 36784/197184 objects degraded (18.655%); 13 KiB/s, 0 objects/s recovering 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: osd.7 (root=default,host=smithi067) is down 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: [WRN] PG_DEGRADED: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-26T21:40:01.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.8 is active+undersized+degraded, acting [5,6] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.11 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.17 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 4.1e is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 5.9 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 5.d is active+undersized+degraded, acting [2,5] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 5.14 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 5.1f is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 6.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.049 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 6.1c is active+undersized+degraded, acting [5,2] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.1 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.3 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.5 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.7 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.9 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.a is active+undersized+degraded, acting [5,1] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.c is active+undersized+degraded, acting [3,4] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.d is active+undersized+degraded, acting [1,5] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[40587]: pg 9.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 36784/197184 objects degraded (18.655%); 13 KiB/s, 0 objects/s recovering 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.050 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: osd.7 (root=default,host=smithi067) is down 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: [WRN] PG_DEGRADED: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.8 is active+undersized+degraded, acting [5,6] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.11 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.051 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.17 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 4.1e is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 5.9 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 5.d is active+undersized+degraded, acting [2,5] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 5.14 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 5.1f is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 6.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 6.1c is active+undersized+degraded, acting [5,2] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.1 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.3 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.5 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.7 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.9 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.a is active+undersized+degraded, acting [5,1] 2024-03-26T21:40:01.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.c is active+undersized+degraded, acting [3,4] 2024-03-26T21:40:01.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.d is active+undersized+degraded, acting [1,5] 2024-03-26T21:40:01.053 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:00 smithi055 ceph-mon[29406]: pg 9.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:01.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 36784/197184 objects degraded (18.655%); 13 KiB/s, 0 objects/s recovering 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: osd.7 (root=default,host=smithi067) is down 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: [WRN] PG_DEGRADED: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-26T21:40:01.149 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.8 is active+undersized+degraded, acting [5,6] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.11 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.17 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 4.1e is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 5.9 is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 5.d is active+undersized+degraded, acting [2,5] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 5.14 is active+undersized+degraded, acting [3,2] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 5.1f is active+undersized+degraded, acting [6,4] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 6.16 is active+undersized+degraded, acting [0,3] 2024-03-26T21:40:01.150 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 6.1c is active+undersized+degraded, acting [5,2] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.1 is active+undersized+degraded, acting [3,6] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.3 is active+undersized+degraded, acting [3,1] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.5 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.7 is active+undersized+degraded, acting [5,3] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.9 is active+undersized+degraded, acting [0,5] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.a is active+undersized+degraded, acting [5,1] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.c is active+undersized+degraded, acting [3,4] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.d is active+undersized+degraded, acting [1,5] 2024-03-26T21:40:01.151 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:00 smithi067 ceph-mon[33346]: pg 9.e is active+undersized+degraded, acting [4,1] 2024-03-26T21:40:01.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.192 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:01.278 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:40:01.293 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:01.332 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:01.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:01.458 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:01.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:01.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:01.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:01.725 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:01.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:01.869 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:01.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:01.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:01.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:40:02.000811+0000 min lat: 0.00115728 max lat: 1.04681 avg lat: 0.00897152 2024-03-26T21:40:01.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:40:01.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 40 16 71326 71310 111.399 74.5 0.0164932 0.00897152 2024-03-26T21:40:02.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[40587]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[40587]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[29406]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T21:40:02.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:02.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:01 smithi055 ceph-mon[29406]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T21:40:02.104 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:01 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:02.104 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:01 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:02.104 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:01 smithi067 ceph-mon[33346]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T21:40:02.104 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:01 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 36784/197184 objects degraded (18.655%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:02.104 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:01 smithi067 ceph-mon[33346]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T21:40:02.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:02.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:02.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:02.293 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:02.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:02.428 INFO:teuthology.orchestra.run.smithi067.stderr:Exporting 4.14 info 4.14( v 234'25 (0'0,234'25] local-lis/les=68/69 n=5 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Read #4:2c0c0c23:reshard::reshard.0000000001:head# 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Read #4:2d09e641:::obj_delete_at_hint.0000000088:head# 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Read #4:2d8f1e3e:::obj_delete_at_hint.0000000080:head# 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Read #4:2dbd613a:::obj_delete_at_hint.0000000090:head# 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Read #4:2ece19cb:::obj_delete_at_hint.0000000110:head# 2024-03-26T21:40:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:Export successful 2024-03-26T21:40:02.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:02.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:02.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:02.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:02.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:02.857 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:02.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:02.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 41 16 72561 72545 110.564 77.1875 0.0153981 0.00903965 2024-03-26T21:40:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:02.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:02.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:03.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[40587]: pgmap v496: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 36784/197184 objects degraded (18.655%) 2024-03-26T21:40:03.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:03.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:03.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[29406]: pgmap v496: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 36784/197184 objects degraded (18.655%) 2024-03-26T21:40:03.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:03.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:03.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:03.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:02 smithi067 ceph-mon[33346]: pgmap v496: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s; 36784/197184 objects degraded (18.655%) 2024-03-26T21:40:03.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:03.110 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:03.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:03.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:03.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:03.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:03.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:03.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:03.664 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:03.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.7 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 4.14 2024-03-26T21:40:03.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:03.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:03.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:03.944 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:03.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:03.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 42 16 73629 73613 109.52 66.75 0.0134341 0.00912589 2024-03-26T21:40:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T21:40:04.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:04.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.222 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:04.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:04.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.363 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:04.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:04.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:04.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:04.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:04.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:04.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:04.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:40:04.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:04.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:04.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:04.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 43 16 74988 74972 108.949 84.9375 0.0122114 0.00917372 2024-03-26T21:40:05.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:05.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:04 smithi055 ceph-mon[29406]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 39953/214899 objects degraded (18.592%) 2024-03-26T21:40:05.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:04 smithi055 ceph-mon[40587]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 39953/214899 objects degraded (18.592%) 2024-03-26T21:40:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:05.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:05.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:05.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:04 smithi067 ceph-mon[33346]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 39953/214899 objects degraded (18.592%) 2024-03-26T21:40:05.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.365 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:05.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:40:05.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:05.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:05.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:05.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.782 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:05.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:05.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:05.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:05.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 44 16 76264 76248 108.285 79.75 0.0279691 0.00922977 2024-03-26T21:40:06.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:06.031 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.070 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:06.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:06.123 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:40:06.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:40:06.167 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:40:06.170 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:06.212 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:40:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:06.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:06.256 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:40:06.293 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:40:06.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:06.332 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:40:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:40:06.387 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:40:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:06.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:06.400 INFO:teuthology.orchestra.run.smithi067.stdout: marking collection for removal 2024-03-26T21:40:06.401 INFO:teuthology.orchestra.run.smithi067.stdout:setting '_remove' omap key 2024-03-26T21:40:06.401 INFO:teuthology.orchestra.run.smithi067.stdout:finish_remove_pgs 4.14_head removing 4.14 2024-03-26T21:40:06.404 INFO:teuthology.orchestra.run.smithi067.stdout:Remove successful 2024-03-26T21:40:06.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:06.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:06.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:06.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:06.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:06.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:06.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:06.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 45 15 77475 77460 107.562 75.75 0.00850145 0.0092918 2024-03-26T21:40:07.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:07.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:07.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:07.171 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: pgmap v499: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 932 op/s; 40317/215991 objects degraded (18.666%) 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1844884842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 40317/215991 objects degraded (18.666%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[29406]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: pgmap v499: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 932 op/s; 40317/215991 objects degraded (18.666%) 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1844884842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:07.172 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 40317/215991 objects degraded (18.666%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:07.173 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:06 smithi055 ceph-mon[40587]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T21:40:07.183 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:40:07.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:40:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: pgmap v499: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 932 op/s; 40317/215991 objects degraded (18.666%) 2024-03-26T21:40:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1844884842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 40317/215991 objects degraded (18.666%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:06 smithi067 ceph-mon[33346]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T21:40:07.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:07.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:07.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.447 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:07.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:07.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:07.585 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:07.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell -n osd.7 --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.14.7 2024-03-26T21:40:07.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:07.728 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:07.830 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:07.869 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:07.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/osd.7/config 2024-03-26T21:40:07.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:07.994 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 46 16 78865 78849 107.109 86.8125 0.00536414 0.00933124 2024-03-26T21:40:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.009 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:08.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:08.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:08.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:08.193 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:08.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:08.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:08.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:08.400 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:08.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:08.635 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:08.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:08.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:08.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:08.918 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:08 smithi055 ceph-mon[29406]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 42612/229890 objects degraded (18.536%) 2024-03-26T21:40:08.918 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:08 smithi055 ceph-mon[40587]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 42612/229890 objects degraded (18.536%) 2024-03-26T21:40:08.966 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:08.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:08.993 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 47 16 80185 80169 106.586 82.5 0.00438908 0.00937664 2024-03-26T21:40:09.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:09.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:09.123 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:08 smithi067 ceph-mon[33346]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 42612/229890 objects degraded (18.536%) 2024-03-26T21:40:09.133 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:40:09.133 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:40:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T21:40:09.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:09.221 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:40:09.237 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.250 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:09.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:09.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:09.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:09.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:09.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:09.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:09.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:09.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:09.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:09.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:09.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:09.994 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 48 16 81460 81444 106.025 79.6875 0.0173293 0.00942723 2024-03-26T21:40:10.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:10.089 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:10.141 INFO:teuthology.orchestra.run.smithi067.stdout:get_pg_num_history pg_num_history pg_num_history(e248 pg_nums {1={19=1},2={66=3},3={66=32},4={68=32},5={70=32},6={72=32},8={173=16},9={240=16}} deleted_pools 236,8) 2024-03-26T21:40:10.142 INFO:teuthology.orchestra.run.smithi067.stdout:Importing pgid 4.14 2024-03-26T21:40:10.142 INFO:teuthology.orchestra.run.smithi067.stdout:Write #4:2c0c0c23:reshard::reshard.0000000001:head# 2024-03-26T21:40:10.142 INFO:teuthology.orchestra.run.smithi067.stdout:snapset 0=[]:{} 2024-03-26T21:40:10.142 INFO:teuthology.orchestra.run.smithi067.stdout:Write #4:2d09e641:::obj_delete_at_hint.0000000088:head# 2024-03-26T21:40:10.142 INFO:teuthology.orchestra.run.smithi067.stdout:snapset 0=[]:{} 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:Write #4:2d8f1e3e:::obj_delete_at_hint.0000000080:head# 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:snapset 0=[]:{} 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:Write #4:2dbd613a:::obj_delete_at_hint.0000000090:head# 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:snapset 0=[]:{} 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:Write #4:2ece19cb:::obj_delete_at_hint.0000000110:head# 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:snapset 0=[]:{} 2024-03-26T21:40:10.143 INFO:teuthology.orchestra.run.smithi067.stdout:write_pg epoch 234 info 4.14( v 234'25 (0'0,234'25] local-lis/les=68/69 n=5 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-26T21:40:10.144 INFO:teuthology.orchestra.run.smithi067.stdout:Import successful 2024-03-26T21:40:10.190 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:10.230 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:10.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:10.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:10.489 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:40:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=cleanup t=2024-03-26T21:40:10.31437589Z level=info msg="Completed cleanup jobs" duration=2.607211ms 2024-03-26T21:40:10.492 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:10.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:10.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:10.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:10.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:10.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:10.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:10.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:40:10.982 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:10 smithi067 ceph-mon[33346]: pgmap v502: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:10.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 49 16 82783 82767 105.547 82.6875 0.0200424 0.0094697 2024-03-26T21:40:11.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:11.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:11.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:11.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:11.221 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:10 smithi055 ceph-mon[29406]: pgmap v502: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:11.221 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:10 smithi055 ceph-mon[40587]: pgmap v502: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:11.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:11.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/exp.4.14.7 2024-03-26T21:40:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:11.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:11.503 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:11.552 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:11.605 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:11.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:11.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:11.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:11.954 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:11.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:11.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 50 16 83726 83710 104.615 58.9375 0.0115365 0.00955323 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[29406]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 43836/234690 objects degraded (18.678%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[29406]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:12.018 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[40587]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T21:40:12.019 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 43836/234690 objects degraded (18.678%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:12.019 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:11 smithi055 ceph-mon[40587]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T21:40:12.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:12.083 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:12.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:11 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:11 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:11 smithi067 ceph-mon[33346]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T21:40:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 43836/234690 objects degraded (18.678%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:11 smithi067 ceph-mon[33346]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T21:40:12.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:12.341 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:12.362 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:12.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:12.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:12.614 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:12.660 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:12.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:12.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:12.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:12.928 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:40:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:12.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:12.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 51 16 84839 84823 103.928 69.5625 0.00594347 0.00961635 2024-03-26T21:40:13.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:13.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[40587]: pgmap v504: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/965442226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[29406]: pgmap v504: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/965442226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:12 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.163 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:13.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:12 smithi067 ceph-mon[33346]: pgmap v504: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s; 43836/234690 objects degraded (18.678%) 2024-03-26T21:40:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/965442226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:12 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:40:13.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:13.333 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:13.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:13.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:13.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:13.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:13.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:13.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:13.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:13.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:13.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:13.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 52 16 86088 86072 103.43 78.0625 0.0165044 0.00966361 2024-03-26T21:40:14.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[29406]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:40:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:13 smithi055 ceph-mon[40587]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T21:40:14.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:14.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:13 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:14.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:13 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:40:14.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:13 smithi067 ceph-mon[33346]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T21:40:14.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:14.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:14.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:14.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:14.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:14.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:14.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:14.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:14.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:14.953 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:14.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:14.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 53 16 87454 87438 103.09 85.375 0.0145438 0.00969532 2024-03-26T21:40:15.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:15.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[40587]: pgmap v507: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s; 46958/252180 objects degraded (18.621%) 2024-03-26T21:40:15.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:40:15.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:40:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[29406]: pgmap v507: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s; 46958/252180 objects degraded (18.621%) 2024-03-26T21:40:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:40:15.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:14 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:40:15.086 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:15.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:15.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:14 smithi067 ceph-mon[33346]: pgmap v507: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s; 46958/252180 objects degraded (18.621%) 2024-03-26T21:40:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:14 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:40:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:14 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:40:15.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:15.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:15.495 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:15.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:15.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:15.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:15.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:15.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:15.938 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:15.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:15.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:40:15.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:40:15.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[40587]: 9.f scrub starts 2024-03-26T21:40:15.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:40:15.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:40:15.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:15 smithi055 ceph-mon[29406]: 9.f scrub starts 2024-03-26T21:40:15.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 54 16 88697 88681 102.619 77.6875 0.0124233 0.00973958 2024-03-26T21:40:16.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:16.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:16.115 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:40:16.168 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:40:16.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:16.207 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:40:16.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:15 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:40:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:15 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:40:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:15 smithi067 ceph-mon[33346]: 9.f scrub starts 2024-03-26T21:40:16.252 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:40:16.289 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:40:16.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:16.336 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:40:16.374 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:16.383 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:40:16.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:40:16.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:40:16.386 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:40:16.387 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:40:16.387 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:40:16.387 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:40:16.387 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:40:16.387 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:40:16.426 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 5, 6, 4] 2024-03-26T21:40:16.426 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:40:16.427 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T21:40:16.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:40:16.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.526 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:16.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:16.627 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:16.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:16.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:16.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:16.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:16.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:16.995 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 55 16 89808 89792 102.016 69.4375 0.00781606 0.00979729 2024-03-26T21:40:17.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: pgmap v508: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s; 47321/253269 objects degraded (18.684%) 2024-03-26T21:40:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:40:17.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2857651320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: 9.f scrub ok 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 47321/253269 objects degraded (18.684%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T21:40:17.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: pgmap v508: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s; 47321/253269 objects degraded (18.684%) 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2857651320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: 9.f scrub ok 2024-03-26T21:40:17.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:40:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:40:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 47321/253269 objects degraded (18.684%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T21:40:17.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:17.072 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:17.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:17.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:17.232 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:40:17.232 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:40:17.232 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:40:17.232 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:40:17.232 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:40:17.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: pgmap v508: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s; 47321/253269 objects degraded (18.684%) 2024-03-26T21:40:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:40:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:40:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2857651320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: 9.f scrub ok 2024-03-26T21:40:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:40:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:40:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 47321/253269 objects degraded (18.684%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T21:40:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:17.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:17.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:40:17.374 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:17.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:17.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:17.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.665 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:17.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:17.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:17.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:17.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:17.961 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:17.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:17.996 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 56 16 91037 91021 101.565 76.8125 0.00528524 0.00984122 2024-03-26T21:40:18.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:18.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:18.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:40:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:17 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:40:18.246 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:40:18.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:40:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:17 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:40:18.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:18.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:18.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:18.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:40:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:18.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:18.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.653 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:18.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:18.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:18.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:40:18.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:40:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:18.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:18.981 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:18.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 57 16 92273 92257 101.137 77.25 0.00769036 0.009883 2024-03-26T21:40:19.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:19.030 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:19.133 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:19.133 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[40587]: pgmap v510: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 49588/267069 objects degraded (18.567%) 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[29406]: pgmap v510: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 49588/267069 objects degraded (18.567%) 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:40:19.134 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:18 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:40:19.155 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:19.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:18 smithi067 ceph-mon[33346]: pgmap v510: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 49588/267069 objects degraded (18.567%) 2024-03-26T21:40:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:18 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:40:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:18 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:40:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:18 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:40:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:18 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:40:19.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.378 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:19.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:19.479 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:19.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:19.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:19.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:19.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:19.795 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:19.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:19.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:19.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 58 16 93400 93384 100.607 70.4375 0.0084538 0.00993503 2024-03-26T21:40:20.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:20.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:20.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:20.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:40:20.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:40:20.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:19 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:40:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:19 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:40:20.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:20.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:20.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:20.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:20.702 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:20.747 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:20.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:20.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:20.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:20.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:40:20.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:20.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 59 16 94684 94668 100.262 80.25 0.0147983 0.00996873 2024-03-26T21:40:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:21.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: pgmap v511: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: pgmap v511: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:40:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:40:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:40:21.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:20 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:40:21.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:21.154 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:21.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: pgmap v511: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:40:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:40:21.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:20 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:40:21.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:21.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:21.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:21.736 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:21.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:21.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:21.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:21.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:21.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:21.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:40:22.005485+0000 min lat: 0.00115728 max lat: 1.04681 avg lat: 0.0100118 2024-03-26T21:40:21.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:40:21.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 60 16 95879 95863 99.8358 74.6875 0.0147816 0.0100118 2024-03-26T21:40:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:22.126 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:22.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:21 smithi067 ceph-mon[33346]: 9.4 scrub starts 2024-03-26T21:40:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:21 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:40:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:21 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:40:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:21 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 50713/271602 objects degraded (18.672%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:22.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:22.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[40587]: 9.4 scrub starts 2024-03-26T21:40:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:40:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:40:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 50713/271602 objects degraded (18.672%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[29406]: 9.4 scrub starts 2024-03-26T21:40:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:40:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:40:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:21 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 50713/271602 objects degraded (18.672%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:22.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:22.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:22.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:22.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:22.816 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:22.852 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:22.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:22.991 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:22.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:22.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 61 16 96996 96980 99.3435 69.8125 0.0257232 0.0100612 2024-03-26T21:40:23.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:23.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[29406]: pgmap v512: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 594 B/s rd, 55 MiB/s wr, 882 op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:40:23.233 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:40:23.234 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[40587]: pgmap v512: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 594 B/s rd, 55 MiB/s wr, 882 op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:23.234 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:40:23.234 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:22 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:40:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:22 smithi067 ceph-mon[33346]: pgmap v512: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 594 B/s rd, 55 MiB/s wr, 882 op/s; 50713/271602 objects degraded (18.672%) 2024-03-26T21:40:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:22 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:40:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:22 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:40:23.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.281 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:23.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:23.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:23.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:23.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:23.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:23.683 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:23.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:23.830 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:23.932 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:23.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:23.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:23.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 62 16 98261 98245 99.0163 79.0625 0.0185522 0.0100945 2024-03-26T21:40:24.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:24.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:24.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:40:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:40:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:40:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:40:24.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:40:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:24 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:40:24.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:24.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:40:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:24 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:40:24.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:24.593 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:24.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:24.742 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:24.844 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:24.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:24.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:24.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 63 16 99521 99505 98.6946 78.75 0.0119433 0.0101279 2024-03-26T21:40:25.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:25.028 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:25.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:25.170 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: 9.4 scrub ok 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: pgmap v513: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 53829/288594 objects degraded (18.652%) 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: 9.4 scrub ok 2024-03-26T21:40:25.260 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: pgmap v513: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 53829/288594 objects degraded (18.652%) 2024-03-26T21:40:25.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:40:25.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:40:25.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:40:25.261 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:25 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:40:25.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:25.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:25.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: 9.4 scrub ok 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: pgmap v513: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 53829/288594 objects degraded (18.652%) 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:40:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:25 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:40:25.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:25.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:25.669 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:25.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:25.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:25.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:25.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:25.955 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:25.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:25.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 64 16 100536 100520 98.1437 63.4375 0.00647583 0.0101759 2024-03-26T21:40:26.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:26.097 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:26.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:26.229 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:26.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:40:26.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:40:26.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:40:26.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:40:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:40:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:40:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:40:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:26 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:40:26.330 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:26.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:26.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:26 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:40:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:26 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:40:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:26 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:40:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:26 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:40:26.505 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:26.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:26.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:26.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:26.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:26.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:26.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:26.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:26.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 65 16 101427 101411 97.4905 55.6875 0.0136525 0.0102526 2024-03-26T21:40:27.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:27.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:27.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:27.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:27.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: pgmap v514: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s; 54214/289749 objects degraded (18.711%) 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 54214/289749 objects degraded (18.711%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[40587]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: pgmap v514: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s; 54214/289749 objects degraded (18.711%) 2024-03-26T21:40:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 54214/289749 objects degraded (18.711%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:27.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:27 smithi055 ceph-mon[29406]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T21:40:27.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:27.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: pgmap v514: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s; 54214/289749 objects degraded (18.711%) 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T21:40:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 54214/289749 objects degraded (18.711%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:27 smithi067 ceph-mon[33346]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T21:40:27.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:27.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:84289 8.1 8:86799a92:::benchmark_data_smithi055_110126_object84288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:02.919744+0000", 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 263.73508633, 2024-03-26T21:40:27.680 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16053445199999999, 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 84289 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919744+0000", 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.681 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919744+0000", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919745+0000", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1629999999999999e-06 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919771+0000", 2024-03-26T21:40:27.682 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.5601000000000001e-05 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919780+0000", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5599999999999999e-06 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.919787+0000", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0659999999999998e-06 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.920653+0000", 2024-03-26T21:40:27.683 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00086550400000000001 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.921421+0000", 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00076829100000000005 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,2", 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.921582+0000", 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016111599999999999 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.684 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.054910+0000", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.13332756200000001 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.057848+0000", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0029378780000000001 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.080166+0000", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.022318056999999999 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.685 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.080229+0000", 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2840000000000001e-05 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.080279+0000", 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9814000000000001e-05 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:84301 8.1 8:8d3ba8b9:::benchmark_data_smithi055_110126_object84300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:02.926720+0000", 2024-03-26T21:40:27.686 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 263.728109948, 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16303894899999999, 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 84301 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.926720+0000", 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.687 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.926720+0000", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.926722+0000", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7940000000000001e-06 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:02.926772+0000", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9308999999999998e-05 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.688 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.053647+0000", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.126875773 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.053672+0000", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4992000000000001e-05 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.054937+0000", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0012645060000000001 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.689 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.055074+0000", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000137081 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,2", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.055235+0000", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016107299999999999 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.058506+0000", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032709290000000001 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.059119+0000", 2024-03-26T21:40:27.690 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000613472 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.089591+0000", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.030471623999999999 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.089706+0000", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000114796 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:03.089759+0000", 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3600000000000002e-05 2024-03-26T21:40:27.691 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:86635 8.7 8:edcd7649:::benchmark_data_smithi055_110126_object86634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:36:06.028413+0000", 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 260.62641738000002, 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.148758054, 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:40:27.692 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 86635 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028413+0000", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028413+0000", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028414+0000", 2024-03-26T21:40:27.693 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.482e-06 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028465+0000", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0226999999999999e-05 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028477+0000", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2274e-05 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028486+0000", 2024-03-26T21:40:27.694 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.568e-06 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028544+0000", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8700999999999999e-05 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028683+0000", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000138753 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.028841+0000", 2024-03-26T21:40:27.695 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015757 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.031970+0000", 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0031297690000000001 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.032896+0000", 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00092548299999999995 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.696 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.176848+0000", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.143951939 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.177087+0000", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000238889 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:36:06.177171+0000", 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4399000000000003e-05 2024-03-26T21:40:27.697 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15096.0:129002 8.7 8:e34d8967:::benchmark_data_smithi055_110126_object15033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:37:02.534285+0000", 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 204.12054504, 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.182228168, 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.698 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15096", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 129002 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534285+0000", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534285+0000", 2024-03-26T21:40:27.699 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534287+0000", 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.863e-06 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534293+0000", 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9950000000000002e-06 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.700 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534306+0000", 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2549e-05 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.534314+0000", 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4700000000000002e-06 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.598281+0000", 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.063967016000000002 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.598391+0000", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109521 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.598513+0000", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121853 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.605110+0000", 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0065972950000000004 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.702 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.608134+0000", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003024568 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.716422+0000", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.10828737200000001 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.716457+0000", 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4832999999999997e-05 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.703 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.704 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:37:02.716514+0000", 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6833000000000003e-05 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.705 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:32058 9.4 9:2c18860a:::benchmark_data_smithi055_135863_object32057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:34.714755+0000", 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 51.94007517, 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.17928596099999999, 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 32058 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.706 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714755+0000", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714755+0000", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714756+0000", 2024-03-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2049999999999999e-06 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714780+0000", 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3189000000000001e-05 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714788+0000", 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1599999999999998e-06 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714793+0000", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4020000000000001e-06 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714866+0000", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2557000000000003e-05 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.714976+0000", 2024-03-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011053900000000001 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.715079+0000", 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010251699999999999 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.716623+0000", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0015437459999999999 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.718271+0000", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016480170000000001 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893955+0000", 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.175684637 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.711 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.894003+0000", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7376999999999997e-05 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.894041+0000", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8615000000000002e-05 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:32079 9.0 9:014155fe:::benchmark_data_smithi055_135863_object32078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:34.723357+0000", 2024-03-26T21:40:27.712 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 51.931473130000001, 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16981945300000001, 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 32079 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723357+0000", 2024-03-26T21:40:27.713 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723357+0000", 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723359+0000", 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.556e-06 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.714 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723384+0000", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4788000000000001e-05 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723396+0000", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2864000000000001e-05 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723406+0000", 2024-03-26T21:40:27.715 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1859999999999998e-06 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723457+0000", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1193999999999998e-05 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723621+0000", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000163809 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,6", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723736+0000", 2024-03-26T21:40:27.716 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115418 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.892832+0000", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.169096102 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.892868+0000", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5746e-05 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893064+0000", 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019568499999999999 2024-03-26T21:40:27.717 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893118+0000", 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4863e-05 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893177+0000", 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8242000000000002e-05 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.718 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:32080 9.e e246/240 9:7c690362:::benchmark_data_smithi055_135863_object32079:head v 246'1936, mlcod=246'1934)", 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:34.723889+0000", 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 51.930941791000002, 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16969896800000001, 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723889+0000", 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.719 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723889+0000", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723889+0000", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7700000000000004e-07 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.723940+0000", 2024-03-26T21:40:27.720 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0556999999999997e-05 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891851+0000", 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.167911697 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891865+0000", 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3447e-05 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.721 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891963+0000", 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8435000000000002e-05 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891990+0000", 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.6342e-05 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.722 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893536+0000", 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0015461749999999999 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.893588+0000", 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1737999999999998e-05 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.25220.0:32079 9.0 e246/240 ondisk, result = 0)", 2024-03-26T21:40:27.723 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:34.724858+0000", 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 51.929972720999999, 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16798171200000001, 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.724858+0000", 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.724 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.724858+0000", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.724858+0000", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6400000000000002e-07 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.724862+0000", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3519999999999998e-06 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891916+0000", 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16705461099999999 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.891925+0000", 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1500000000000005e-06 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.892819+0000", 2024-03-26T21:40:27.726 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00089391099999999995 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.892827+0000", 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4710000000000001e-06 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:34.892839+0000", 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2653000000000001e-05 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.727 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:34144 9.0 9:0439d7a7:::benchmark_data_smithi055_135863_object34143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.269413+0000", 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.385417234000002, 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14864665599999999, 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 34144 2024-03-26T21:40:27.728 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269413+0000", 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269413+0000", 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.729 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269415+0000", 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.381e-06 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269434+0000", 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9494999999999999e-05 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269443+0000", 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0049999999999993e-06 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.730 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269448+0000", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3900000000000001e-06 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269518+0000", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9691999999999996e-05 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269640+0000", 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121993 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.731 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,6", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269728+0000", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7807999999999997e-05 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416762+0000", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14703428099999999 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416803+0000", 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0469999999999997e-05 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.732 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417950+0000", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001147716 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.418012+0000", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1280999999999998e-05 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.418060+0000", 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8143999999999998e-05 2024-03-26T21:40:27.733 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:34143 9.6 e246/240 9:6e3293d0:::benchmark_data_smithi055_135863_object34142:head v 246'2227, mlcod=246'2225)", 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.269892+0000", 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.384938282, 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.156694634, 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.734 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269892+0000", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269892+0000", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269893+0000", 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5499999999999998e-07 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.735 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269926+0000", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3395000000000001e-05 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269931+0000", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.395e-06 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269935+0000", 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8190000000000002e-06 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.736 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.269993+0000", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7744000000000001e-05 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270011+0000", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8196e-05 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.426539+0000", 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.156528311 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.737 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.426587+0000", 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7318999999999999e-05 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:34151 9.0 9:06071b55:::benchmark_data_smithi055_135863_object34150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.270177+0000", 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.384653739000001, 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14797796599999999, 2024-03-26T21:40:27.738 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 34151 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270177+0000", 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.739 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270177+0000", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270178+0000", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7199999999999997e-07 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270198+0000", 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.048e-05 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.740 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270209+0000", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0638e-05 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.270216+0000", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5120000000000002e-06 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271024+0000", 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00080765199999999996 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.741 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271172+0000", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014850700000000001 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,6", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271277+0000", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104801 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417288+0000", 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14601042 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.742 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417321+0000", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3099999999999998e-05 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.418071+0000", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00075001500000000003 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.418113+0000", 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1811999999999999e-05 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.743 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.418155+0000", 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2057000000000001e-05 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.25220.0:34144 9.0 e246/240 ondisk, result = 0)", 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.271488+0000", 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.383342044000003, 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14528227399999999, 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.744 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271488+0000", 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271488+0000", 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.745 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271489+0000", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2900000000000004e-07 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271492+0000", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.619e-06 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271500+0000", 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9249999999999995e-06 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.746 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271506+0000", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9449999999999998e-06 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416745+0000", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14523835500000001 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416755+0000", 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0186999999999999e-05 2024-03-26T21:40:27.747 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416771+0000", 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5713999999999999e-05 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.25220.0:34144 9.0 e246/240 ondisk, result = 0)", 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.271538+0000", 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.383292777000001, 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14527543100000001, 2024-03-26T21:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271538+0000", 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271538+0000", 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.749 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271538+0000", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9500000000000003e-07 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271541+0000", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0960000000000001e-06 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271564+0000", 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2690000000000001e-05 2024-03-26T21:40:27.750 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271578+0000", 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3648999999999999e-05 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416791+0000", 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.145213756 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.751 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416798+0000", 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5740000000000001e-06 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416813+0000", 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5170999999999999e-05 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.752 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:34147 9.e e246/240 9:71e4b65e:::benchmark_data_smithi055_135863_object34146:head v 246'2052, mlcod=246'2049)", 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.271577+0000", 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.383253615000001, 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15507164100000001, 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271577+0000", 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.753 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271577+0000", 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271577+0000", 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9900000000000001e-07 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.754 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271624+0000", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6845999999999998e-05 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271632+0000", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2730000000000002e-06 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271638+0000", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0399999999999998e-06 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.755 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271682+0000", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3967000000000001e-05 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.271699+0000", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7071e-05 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.426593+0000", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15489312399999999 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.756 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.426648+0000", 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5821e-05 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:34145 9.a e246/240 9:513169cf:::benchmark_data_smithi055_135863_object34144:head v 246'2158, mlcod=246'2155)", 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.272126+0000", 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.382704273999998, 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14955215399999999, 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.757 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.272126+0000", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.272126+0000", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.272127+0000", 2024-03-26T21:40:27.758 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9999999999999998e-07 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.272189+0000", 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2901e-05 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416982+0000", 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14479251300000001 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.759 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.416992+0000", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0379e-05 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417050+0000", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7550999999999998e-05 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417072+0000", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2004000000000001e-05 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.760 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.421095+0000", 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040231920000000001 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.421678+0000", 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00058321399999999998 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:34150 9.e e246/240 9:79340bf3:::benchmark_data_smithi055_135863_object34149:head v 246'2053, mlcod=246'2049)", 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:39:36.274115+0000", 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 50.380715078000001, 2024-03-26T21:40:27.761 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15317112999999999, 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.274115+0000", 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.274115+0000", 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.274116+0000", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7999999999999995e-07 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.274163+0000", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7296000000000003e-05 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417356+0000", 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.14319278399999999 2024-03-26T21:40:27.763 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417363+0000", 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8260000000000004e-06 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417791+0000", 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000428472 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.764 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.417811+0000", 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9973e-05 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.427258+0000", 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0094464479999999997 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:39:36.427286+0000", 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.8751e-05 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.765 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:72801 9.4 9:2ef81286:::benchmark_data_smithi055_135863_object72800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:40:03.171142+0000", 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 23.483688268000002, 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.21580450800000001, 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 72801 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.766 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171142+0000", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171142+0000", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171143+0000", 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3600000000000002e-07 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.767 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171171+0000", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7713999999999999e-05 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171180+0000", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5380000000000008e-06 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.171187+0000", 2024-03-26T21:40:27.768 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.233e-06 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175065+0000", 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038773779999999999 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175230+0000", 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016526799999999999 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175394+0000", 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000163967 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.219445+0000", 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.044051356 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.219493+0000", 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7979000000000003e-05 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.770 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.386845+0000", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16735129600000001 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.386903+0000", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8056000000000001e-05 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.386947+0000", 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3887000000000002e-05 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.771 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:72807 9.4 9:20abd783:::benchmark_data_smithi055_135863_object72806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:40:03.175146+0000", 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 23.479684625000001, 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.21191427299999999, 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 72807 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.772 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175146+0000", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175146+0000", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175154+0000", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1219999999999995e-06 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.773 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175179+0000", 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.5551999999999999e-05 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175188+0000", 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2700000000000004e-06 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.175193+0000", 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3909999999999997e-06 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.774 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.216728+0000", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.041534806000000001 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.216873+0000", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014525200000000001 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.217029+0000", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015627300000000001 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.775 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.220094+0000", 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030645799999999999 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.220227+0000", 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013267199999999999 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.386984+0000", 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.166757708 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.776 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.387027+0000", 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3000000000000002e-05 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.387060+0000", 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2647000000000001e-05 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:72818 9.4 9:284b4f92:::benchmark_data_smithi055_135863_object72817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-26T21:40:27.777 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:40:03.223785+0000", 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 23.431045495999999, 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.172177213, 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 72818 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.778 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223785+0000", 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223785+0000", 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.779 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223786+0000", 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2929999999999999e-06 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223809+0000", 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3315999999999998e-05 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223822+0000", 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2014000000000001e-05 2024-03-26T21:40:27.780 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223830+0000", 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.619e-06 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.223895+0000", 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5086999999999995e-05 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.781 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.224110+0000", 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000215217 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.224290+0000", 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000179798 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.226968+0000", 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0026780839999999998 2024-03-26T21:40:27.782 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.227478+0000", 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00050952699999999996 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.395860+0000", 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.16838225900000001 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.783 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.395920+0000", 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9549999999999997e-05 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.395962+0000", 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2449e-05 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.784 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:72832 9.4 9:2d4db778:::benchmark_data_smithi055_135863_object72831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:40:03.236312+0000", 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 23.418518807000002, 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15980006599999999, 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 72832 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:40:27.785 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236312+0000", 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236312+0000", 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:40:27.786 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236313+0000", 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2440000000000001e-06 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236332+0000", 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9613999999999999e-05 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236343+0000", 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0396e-05 2024-03-26T21:40:27.787 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236349+0000", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0639999999999999e-06 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236395+0000", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5827e-05 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236487+0000", 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2410999999999993e-05 2024-03-26T21:40:27.788 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.236581+0000", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4313000000000003e-05 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.238530+0000", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0019488179999999999 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.238848+0000", 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00031726500000000001 2024-03-26T21:40:27.789 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.396003+0000", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.15715564300000001 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.396075+0000", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1517e-05 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:40:03.396112+0000", 2024-03-26T21:40:27.790 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6953999999999999e-05 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:40:27.791 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:40:27.791 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:27.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:27.916 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:27.955 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:27.997 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 66 16 102474 102458 97.0048 65.4375 0.0117875 0.0103042 2024-03-26T21:40:28.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:28.080 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:28.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:28.211 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:28.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:28.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:40:28.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:40:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:40:28.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:28.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:28.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:40:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:28 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:40:28.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:28.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:28.726 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:28.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:28.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:28.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:28.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:28.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 67 16 103647 103631 96.6511 73.3125 0.00736792 0.0103419 2024-03-26T21:40:29.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:29.047 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:29.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:29.178 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:29.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:29.307 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:29 smithi055 ceph-mon[40587]: pgmap v517: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 56319/302382 objects degraded (18.625%) 2024-03-26T21:40:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:29 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:40:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:29 smithi055 ceph-mon[29406]: pgmap v517: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 56319/302382 objects degraded (18.625%) 2024-03-26T21:40:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:29 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:40:29.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:29.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.455 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:29 smithi067 ceph-mon[33346]: pgmap v517: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 56319/302382 objects degraded (18.625%) 2024-03-26T21:40:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:29 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:40:29.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:29.588 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:29.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:29.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:29.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:29.856 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:29.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:29.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:29.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 68 16 104815 104799 96.3032 73 0.012654 0.0103794 2024-03-26T21:40:30.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:30.118 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:30.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:30.250 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:30.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:30 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:40:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:30 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:40:30.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:30.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:30 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:40:30.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:30.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:30.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:30.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.782 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:30.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:30.883 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:30.912 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:30.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:30.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 69 16 105980 105964 95.9627 72.8125 0.012085 0.0104164 2024-03-26T21:40:31.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:31.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:31.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:31.283 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[29406]: pgmap v518: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[40587]: pgmap v518: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:40:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:31 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:40:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:31.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:31.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:31.465 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:31 smithi067 ceph-mon[33346]: pgmap v518: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:31 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:40:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:31 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:40:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:31 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:40:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:31 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:40:31.567 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:31.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:31.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:31.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:31.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:31.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:31.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 70 16 107166 107150 95.6508 74.125 0.00955302 0.0104494 2024-03-26T21:40:31.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:31.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:32.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:32.130 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:32.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:32.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 57390/306582 objects degraded (18.719%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:32.238 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:32.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 57390/306582 objects degraded (18.719%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:32.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:32.275 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:32.376 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:32.414 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 57390/306582 objects degraded (18.719%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:32.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:32.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:32.685 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:32.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:32.829 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:32.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:40:32.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:32.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:32.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:32.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 71 16 108348 108332 95.344 73.875 0.0100878 0.0104834 2024-03-26T21:40:33.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:33.086 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:33.095 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:33.118 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:40:33.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.250 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:33.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:33.351 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.416 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:33.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[29406]: pgmap v519: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/959910363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[40587]: pgmap v519: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:33.487 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:40:33.488 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:40:33.488 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/959910363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.488 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:33 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:33 smithi067 ceph-mon[33346]: pgmap v519: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s; 57390/306582 objects degraded (18.719%) 2024-03-26T21:40:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:33 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:40:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:33 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:40:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/959910363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:33 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:40:33.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:33.528 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:40:33.536 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:40:33.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:33.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:33.676 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:33.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:33.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:33.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:33.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 72 16 109335 109319 94.8765 61.6875 0.0209789 0.010535 2024-03-26T21:40:34.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:34.096 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:34.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:34.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:34.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:34.376 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[29406]: 9.0 scrub starts 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[29406]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[40587]: 9.0 scrub starts 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:40:34.377 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:34 smithi055 ceph-mon[40587]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T21:40:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:34.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:34.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:34 smithi067 ceph-mon[33346]: 9.0 scrub starts 2024-03-26T21:40:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:34 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:40:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:40:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:34 smithi067 ceph-mon[33346]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T21:40:34.535 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:34.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:34.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:34.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:34.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:34.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:34.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:34.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:34.984 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-03-26T21:40:33.106401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6957242792,"num_objects":106348,"num_object_clones":0,"num_object_copies":319044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106348,"num_whiteouts":0,"num_read":2036,"num_read_kb":1884,"num_write":107051,"num_write_kb":6796061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1971981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83021,"ondisk_log_size":83021,"up":382,"acting":382,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21317264,"kb_used_data":18304204,"kb_used_omap":15,"kb_used_meta":3012976,"kb_avail":728643952,"statfs":{"total":767960285184,"available":746131406848,"internally_reserved":0,"allocated":18743504896,"data_stored":18739765406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":3085288206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,3,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":639959040,"num_objects":9765,"num_object_clones":0,"num_object_copies":29295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9765,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9765,"num_write_kb":624960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.579570"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726014+0000","last_change":"2024-03-26T21:39:45.902565+0000","last_active":"2024-03-26T21:40:26.726014+0000","last_peered":"2024-03-26T21:40:26.726014+0000","last_clean":"2024-03-26T21:40:26.726014+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:40:26.726014+0000","last_undegraded":"2024-03-26T21:40:26.726014+0000","last_fullsized":"2024-03-26T21:40:26.726014+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:45.902454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.902454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:07.848765+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":444,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.735399+0000","last_change":"2024-03-26T21:39:47.632927+0000","last_active":"2024-03-26T21:40:26.735399+0000","last_peered":"2024-03-26T21:40:26.735399+0000","last_clean":"2024-03-26T21:39:42.855190+0000","last_became_active":"2024-03-26T21:39:47.632927+0000","last_became_peered":"2024-03-26T21:39:47.632927+0000","last_unstale":"2024-03-26T21:40:26.735399+0000","last_undegraded":"2024-03-26T21:40:26.735399+0000","last_fullsized":"2024-03-26T21:39:47.615716+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:05.806266+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.742904+0000","last_change":"2024-03-26T21:40:20.651612+0000","last_active":"2024-03-26T21:40:26.742904+0000","last_peered":"2024-03-26T21:40:26.742904+0000","last_clean":"2024-03-26T21:40:26.742904+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:40:26.742904+0000","last_undegraded":"2024-03-26T21:40:26.742904+0000","last_fullsized":"2024-03-26T21:40:26.742904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.651494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.651494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:43.042937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":157,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.943984+0000","last_change":"2024-03-26T21:40:26.944039+0000","last_active":"2024-03-26T21:40:26.943984+0000","last_peered":"2024-03-26T21:40:26.943984+0000","last_clean":"2024-03-26T21:40:26.943984+0000","last_became_active":"2024-03-26T21:36:51.695519+0000","last_became_peered":"2024-03-26T21:36:51.695519+0000","last_unstale":"2024-03-26T21:40:26.943984+0000","last_undegraded":"2024-03-26T21:40:26.943984+0000","last_fullsized":"2024-03-26T21:40:26.943984+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:40:26.943893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:26.943893+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:56.125834+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.747143+0000","last_change":"2024-03-26T21:39:47.629714+0000","last_active":"2024-03-26T21:40:26.747143+0000","last_peered":"2024-03-26T21:40:26.747143+0000","last_clean":"2024-03-26T21:39:37.770496+0000","last_became_active":"2024-03-26T21:39:47.629714+0000","last_became_peered":"2024-03-26T21:39:47.629714+0000","last_unstale":"2024-03-26T21:40:26.747143+0000","last_undegraded":"2024-03-26T21:40:26.747143+0000","last_fullsized":"2024-03-26T21:39:47.616156+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:53.766318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:37:53.766318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.725419+0000","last_change":"2024-03-26T21:39:38.803536+0000","last_active":"2024-03-26T21:40:26.725419+0000","last_peered":"2024-03-26T21:40:26.725419+0000","last_clean":"2024-03-26T21:40:26.725419+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:40:26.725419+0000","last_undegraded":"2024-03-26T21:40:26.725419+0000","last_fullsized":"2024-03-26T21:40:26.725419+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:38.803402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:38.803402+0000","last_clean_scrub_stamp":"2024-03-26T21:39:38.803402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:42.462185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.743432+0000","last_change":"2024-03-26T21:37:41.557249+0000","last_active":"2024-03-26T21:40:26.743432+0000","last_peered":"2024-03-26T21:40:26.743432+0000","last_clean":"2024-03-26T21:40:26.743432+0000","last_became_active":"2024-03-26T21:28:34.840113+0000","last_became_peered":"2024-03-26T21:28:34.840113+0000","last_unstale":"2024-03-26T21:40:26.743432+0000","last_undegraded":"2024-03-26T21:40:26.743432+0000","last_fullsized":"2024-03-26T21:40:26.743432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:41.557115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:41.557115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":472,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.746746+0000","last_change":"2024-03-26T21:38:46.990301+0000","last_active":"2024-03-26T21:40:26.746746+0000","last_peered":"2024-03-26T21:40:26.746746+0000","last_clean":"2024-03-26T21:40:26.746746+0000","last_became_active":"2024-03-26T21:28:36.846015+0000","last_became_peered":"2024-03-26T21:28:36.846015+0000","last_unstale":"2024-03-26T21:40:26.746746+0000","last_undegraded":"2024-03-26T21:40:26.746746+0000","last_fullsized":"2024-03-26T21:40:26.746746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:38:46.990173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.990173+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.746759+0000","last_change":"2024-03-26T21:39:47.633749+0000","last_active":"2024-03-26T21:40:26.746759+0000","last_peered":"2024-03-26T21:40:26.746759+0000","last_clean":"2024-03-26T21:39:37.738449+0000","last_became_active":"2024-03-26T21:39:47.633749+0000","last_became_peered":"2024-03-26T21:39:47.633749+0000","last_unstale":"2024-03-26T21:40:26.746759+0000","last_undegraded":"2024-03-26T21:40:26.746759+0000","last_fullsized":"2024-03-26T21:39:47.612293+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.492302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.492302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":436,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.738792+0000","last_change":"2024-03-26T21:39:47.621366+0000","last_active":"2024-03-26T21:40:26.738792+0000","last_peered":"2024-03-26T21:40:26.738792+0000","last_clean":"2024-03-26T21:39:39.192192+0000","last_became_active":"2024-03-26T21:39:47.621366+0000","last_became_peered":"2024-03-26T21:39:47.621366+0000","last_unstale":"2024-03-26T21:40:26.738792+0000","last_undegraded":"2024-03-26T21:40:26.738792+0000","last_fullsized":"2024-03-26T21:39:47.610995+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.192065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.192065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:01.176015+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":454,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738719+0000","last_change":"2024-03-26T21:38:48.846089+0000","last_active":"2024-03-26T21:40:26.738719+0000","last_peered":"2024-03-26T21:40:26.738719+0000","last_clean":"2024-03-26T21:40:26.738719+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:40:26.738719+0000","last_undegraded":"2024-03-26T21:40:26.738719+0000","last_fullsized":"2024-03-26T21:40:26.738719+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:48.845955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:38:48.845955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":157,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726290+0000","last_change":"2024-03-26T21:39:44.827305+0000","last_active":"2024-03-26T21:40:26.726290+0000","last_peered":"2024-03-26T21:40:26.726290+0000","last_clean":"2024-03-26T21:40:26.726290+0000","last_became_active":"2024-03-26T21:36:51.703479+0000","last_became_peered":"2024-03-26T21:36:51.703479+0000","last_unstale":"2024-03-26T21:40:26.726290+0000","last_undegraded":"2024-03-26T21:40:26.726290+0000","last_fullsized":"2024-03-26T21:40:26.726290+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:44.827161+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:39:44.827161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:07.424326+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":429,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738023+0000","last_change":"2024-03-26T21:39:25.211203+0000","last_active":"2024-03-26T21:40:26.738023+0000","last_peered":"2024-03-26T21:40:26.738023+0000","last_clean":"2024-03-26T21:40:26.738023+0000","last_became_active":"2024-03-26T21:28:40.886718+0000","last_became_peered":"2024-03-26T21:28:40.886718+0000","last_unstale":"2024-03-26T21:40:26.738023+0000","last_undegraded":"2024-03-26T21:40:26.738023+0000","last_fullsized":"2024-03-26T21:40:26.738023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:25.211059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.211059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:29.098655+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.725801+0000","last_change":"2024-03-26T21:39:47.630921+0000","last_active":"2024-03-26T21:40:26.725801+0000","last_peered":"2024-03-26T21:40:26.725801+0000","last_clean":"2024-03-26T21:36:43.772604+0000","last_became_active":"2024-03-26T21:39:47.630921+0000","last_became_peered":"2024-03-26T21:39:47.630921+0000","last_unstale":"2024-03-26T21:40:26.725801+0000","last_undegraded":"2024-03-26T21:40:26.725801+0000","last_fullsized":"2024-03-26T21:39:47.605619+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:28.958626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:28.958626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:35.070615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":454,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738484+0000","last_change":"2024-03-26T21:38:45.911736+0000","last_active":"2024-03-26T21:40:26.738484+0000","last_peered":"2024-03-26T21:40:26.738484+0000","last_clean":"2024-03-26T21:40:26.738484+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:40:26.738484+0000","last_undegraded":"2024-03-26T21:40:26.738484+0000","last_fullsized":"2024-03-26T21:40:26.738484+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:45.911638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:45.911638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":139,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726458+0000","last_change":"2024-03-26T21:39:28.766370+0000","last_active":"2024-03-26T21:40:26.726458+0000","last_peered":"2024-03-26T21:40:26.726458+0000","last_clean":"2024-03-26T21:40:26.726458+0000","last_became_active":"2024-03-26T21:36:51.700845+0000","last_became_peered":"2024-03-26T21:36:51.700845+0000","last_unstale":"2024-03-26T21:40:26.726458+0000","last_undegraded":"2024-03-26T21:40:26.726458+0000","last_fullsized":"2024-03-26T21:40:26.726458+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:39:28.766237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:28.766237+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:41.122988+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":130,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.730098+0000","last_change":"2024-03-26T21:39:23.811168+0000","last_active":"2024-03-26T21:40:26.730098+0000","last_peered":"2024-03-26T21:40:26.730098+0000","last_clean":"2024-03-26T21:40:26.730098+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:40:26.730098+0000","last_undegraded":"2024-03-26T21:40:26.730098+0000","last_fullsized":"2024-03-26T21:40:26.730098+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.811041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.811041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.648744+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":462,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.733449+0000","last_change":"2024-03-26T21:40:16.881228+0000","last_active":"2024-03-26T21:40:26.733449+0000","last_peered":"2024-03-26T21:40:26.733449+0000","last_clean":"2024-03-26T21:40:26.733449+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:40:26.733449+0000","last_undegraded":"2024-03-26T21:40:26.733449+0000","last_fullsized":"2024-03-26T21:40:26.733449+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.881096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:41.289154+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":449,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.741701+0000","last_change":"2024-03-26T21:39:47.615439+0000","last_active":"2024-03-26T21:40:26.741701+0000","last_peered":"2024-03-26T21:40:26.741701+0000","last_clean":"2024-03-26T21:39:37.766862+0000","last_became_active":"2024-03-26T21:39:47.615439+0000","last_became_peered":"2024-03-26T21:39:47.615439+0000","last_unstale":"2024-03-26T21:40:26.741701+0000","last_undegraded":"2024-03-26T21:40:26.741701+0000","last_fullsized":"2024-03-26T21:39:47.609466+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:42.479878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:42.479878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":496,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.751221+0000","last_change":"2024-03-26T21:40:24.260690+0000","last_active":"2024-03-26T21:40:26.751221+0000","last_peered":"2024-03-26T21:40:26.751221+0000","last_clean":"2024-03-26T21:40:26.751221+0000","last_became_active":"2024-03-26T21:28:36.841314+0000","last_became_peered":"2024-03-26T21:28:36.841314+0000","last_unstale":"2024-03-26T21:40:26.751221+0000","last_undegraded":"2024-03-26T21:40:26.751221+0000","last_fullsized":"2024-03-26T21:40:26.751221+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:24.260540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:24.260540+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:42.669879+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.755554+0000","last_change":"2024-03-26T21:40:17.347214+0000","last_active":"2024-03-26T21:40:26.755554+0000","last_peered":"2024-03-26T21:40:26.755554+0000","last_clean":"2024-03-26T21:40:26.755554+0000","last_became_active":"2024-03-26T21:28:41.113543+0000","last_became_peered":"2024-03-26T21:28:41.113543+0000","last_unstale":"2024-03-26T21:40:26.755554+0000","last_undegraded":"2024-03-26T21:40:26.755554+0000","last_fullsized":"2024-03-26T21:40:26.755554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.347011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.347011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:28.097697+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726228+0000","last_change":"2024-03-26T21:39:42.848155+0000","last_active":"2024-03-26T21:40:26.726228+0000","last_peered":"2024-03-26T21:40:26.726228+0000","last_clean":"2024-03-26T21:40:26.726228+0000","last_became_active":"2024-03-26T21:36:51.700623+0000","last_became_peered":"2024-03-26T21:36:51.700623+0000","last_unstale":"2024-03-26T21:40:26.726228+0000","last_undegraded":"2024-03-26T21:40:26.726228+0000","last_fullsized":"2024-03-26T21:40:26.726228+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.847981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:51.557609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":130,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.725917+0000","last_change":"2024-03-26T21:38:42.881873+0000","last_active":"2024-03-26T21:40:26.725917+0000","last_peered":"2024-03-26T21:40:26.725917+0000","last_clean":"2024-03-26T21:40:26.725917+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:40:26.725917+0000","last_undegraded":"2024-03-26T21:40:26.725917+0000","last_fullsized":"2024-03-26T21:40:26.725917+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:42.881796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:42.881796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":490,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.746315+0000","last_change":"2024-03-26T21:36:51.825498+0000","last_active":"2024-03-26T21:40:26.746315+0000","last_peered":"2024-03-26T21:40:26.746315+0000","last_clean":"2024-03-26T21:40:26.746315+0000","last_became_active":"2024-03-26T21:36:51.690591+0000","last_became_peered":"2024-03-26T21:36:51.690591+0000","last_unstale":"2024-03-26T21:40:26.746315+0000","last_undegraded":"2024-03-26T21:40:26.746315+0000","last_fullsized":"2024-03-26T21:40:26.746315+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-26T21:36:42.463060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:36:42.463060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":41,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.738237+0000","last_change":"2024-03-26T21:39:47.618992+0000","last_active":"2024-03-26T21:40:26.738237+0000","last_peered":"2024-03-26T21:40:26.738237+0000","last_clean":"2024-03-26T21:28:41.897055+0000","last_became_active":"2024-03-26T21:39:47.618992+0000","last_became_peered":"2024-03-26T21:39:47.618992+0000","last_unstale":"2024-03-26T21:40:26.738237+0000","last_undegraded":"2024-03-26T21:39:47.610283+0000","last_fullsized":"2024-03-26T21:39:47.610170+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:38:56.062344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:56.062344+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"250'13","reported_seq":625,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.733372+0000","last_change":"2024-03-26T21:39:47.619315+0000","last_active":"2024-03-26T21:40:26.733372+0000","last_peered":"2024-03-26T21:40:26.733372+0000","last_clean":"2024-03-26T21:39:45.023660+0000","last_became_active":"2024-03-26T21:39:47.619315+0000","last_became_peered":"2024-03-26T21:39:47.619315+0000","last_unstale":"2024-03-26T21:40:26.733372+0000","last_undegraded":"2024-03-26T21:39:47.612416+0000","last_fullsized":"2024-03-26T21:39:47.612292+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:38:48.015374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:38:48.015374+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:31.943477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":480,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.735739+0000","last_change":"2024-03-26T21:40:23.863029+0000","last_active":"2024-03-26T21:40:26.735739+0000","last_peered":"2024-03-26T21:40:26.735739+0000","last_clean":"2024-03-26T21:40:26.735739+0000","last_became_active":"2024-03-26T21:36:51.709840+0000","last_became_peered":"2024-03-26T21:36:51.709840+0000","last_unstale":"2024-03-26T21:40:26.735739+0000","last_undegraded":"2024-03-26T21:40:26.735739+0000","last_fullsized":"2024-03-26T21:40:26.735739+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:40:23.862904+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.862904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:30.893504+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":463,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.742059+0000","last_change":"2024-03-26T21:39:47.628451+0000","last_active":"2024-03-26T21:40:26.742059+0000","last_peered":"2024-03-26T21:40:26.742059+0000","last_clean":"2024-03-26T21:39:37.766117+0000","last_became_active":"2024-03-26T21:39:47.628451+0000","last_became_peered":"2024-03-26T21:39:47.628451+0000","last_unstale":"2024-03-26T21:40:26.742059+0000","last_undegraded":"2024-03-26T21:39:47.610723+0000","last_fullsized":"2024-03-26T21:39:47.608385+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:39:33.305513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.305513+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:41.759957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":458,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.733127+0000","last_change":"2024-03-26T21:40:16.025952+0000","last_active":"2024-03-26T21:40:26.733127+0000","last_peered":"2024-03-26T21:40:26.733127+0000","last_clean":"2024-03-26T21:40:26.733127+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:40:26.733127+0000","last_undegraded":"2024-03-26T21:40:26.733127+0000","last_fullsized":"2024-03-26T21:40:26.733127+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.025838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.025838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:21.239727+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":429,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.746521+0000","last_change":"2024-03-26T21:39:47.625683+0000","last_active":"2024-03-26T21:40:26.746521+0000","last_peered":"2024-03-26T21:40:26.746521+0000","last_clean":"2024-03-26T21:39:37.766384+0000","last_became_active":"2024-03-26T21:39:47.625683+0000","last_became_peered":"2024-03-26T21:39:47.625683+0000","last_unstale":"2024-03-26T21:40:26.746521+0000","last_undegraded":"2024-03-26T21:40:26.746521+0000","last_fullsized":"2024-03-26T21:39:47.611329+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:51.775881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.775881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.742509+0000","last_change":"2024-03-26T21:39:24.208347+0000","last_active":"2024-03-26T21:40:26.742509+0000","last_peered":"2024-03-26T21:40:26.742509+0000","last_clean":"2024-03-26T21:40:26.742509+0000","last_became_active":"2024-03-26T21:28:34.848944+0000","last_became_peered":"2024-03-26T21:28:34.848944+0000","last_unstale":"2024-03-26T21:40:26.742509+0000","last_undegraded":"2024-03-26T21:40:26.742509+0000","last_fullsized":"2024-03-26T21:40:26.742509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:24.208179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.208179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:53.314633+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":490,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.947811+0000","last_change":"2024-03-26T21:40:26.947869+0000","last_active":"2024-03-26T21:40:26.947811+0000","last_peered":"2024-03-26T21:40:26.947811+0000","last_clean":"2024-03-26T21:40:26.947811+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:40:26.947811+0000","last_undegraded":"2024-03-26T21:40:26.947811+0000","last_fullsized":"2024-03-26T21:40:26.947811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:26.947732+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:40:26.947732+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:46.517823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":343,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.751711+0000","last_change":"2024-03-26T21:39:47.613284+0000","last_active":"2024-03-26T21:40:26.751711+0000","last_peered":"2024-03-26T21:40:26.751711+0000","last_clean":"2024-03-26T21:36:43.772505+0000","last_became_active":"2024-03-26T21:39:47.613284+0000","last_became_peered":"2024-03-26T21:39:47.613284+0000","last_unstale":"2024-03-26T21:40:26.751711+0000","last_undegraded":"2024-03-26T21:40:26.751711+0000","last_fullsized":"2024-03-26T21:39:47.610569+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:54.032885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:54.032885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:29.249596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":458,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.732873+0000","last_change":"2024-03-26T21:39:47.608749+0000","last_active":"2024-03-26T21:40:26.732873+0000","last_peered":"2024-03-26T21:40:26.732873+0000","last_clean":"2024-03-26T21:39:37.768118+0000","last_became_active":"2024-03-26T21:39:47.608749+0000","last_became_peered":"2024-03-26T21:39:47.608749+0000","last_unstale":"2024-03-26T21:40:26.732873+0000","last_undegraded":"2024-03-26T21:40:26.732873+0000","last_fullsized":"2024-03-26T21:39:47.607338+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:45.982479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:38:45.982479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:27.990206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":441,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.743123+0000","last_change":"2024-03-26T21:39:29.295033+0000","last_active":"2024-03-26T21:40:26.743123+0000","last_peered":"2024-03-26T21:40:26.743123+0000","last_clean":"2024-03-26T21:40:26.743123+0000","last_became_active":"2024-03-26T21:28:34.843647+0000","last_became_peered":"2024-03-26T21:28:34.843647+0000","last_unstale":"2024-03-26T21:40:26.743123+0000","last_undegraded":"2024-03-26T21:40:26.743123+0000","last_fullsized":"2024-03-26T21:40:26.743123+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:29.294907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.294907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:56.147227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.742617+0000","last_change":"2024-03-26T21:37:48.600196+0000","last_active":"2024-03-26T21:40:26.742617+0000","last_peered":"2024-03-26T21:40:26.742617+0000","last_clean":"2024-03-26T21:40:26.742617+0000","last_became_active":"2024-03-26T21:28:36.840516+0000","last_became_peered":"2024-03-26T21:28:36.840516+0000","last_unstale":"2024-03-26T21:40:26.742617+0000","last_undegraded":"2024-03-26T21:40:26.742617+0000","last_fullsized":"2024-03-26T21:40:26.742617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-26T21:37:48.600034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:37:48.600034+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.747320+0000","last_change":"2024-03-26T21:39:36.269080+0000","last_active":"2024-03-26T21:40:26.747320+0000","last_peered":"2024-03-26T21:40:26.747320+0000","last_clean":"2024-03-26T21:40:26.747320+0000","last_became_active":"2024-03-26T21:28:41.117346+0000","last_became_peered":"2024-03-26T21:28:41.117346+0000","last_unstale":"2024-03-26T21:40:26.747320+0000","last_undegraded":"2024-03-26T21:40:26.747320+0000","last_fullsized":"2024-03-26T21:40:26.747320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:36.268946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:39:36.268946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:45.147653+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":339,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.725497+0000","last_change":"2024-03-26T21:39:47.608473+0000","last_active":"2024-03-26T21:40:26.725497+0000","last_peered":"2024-03-26T21:40:26.725497+0000","last_clean":"2024-03-26T21:36:43.772657+0000","last_became_active":"2024-03-26T21:39:47.608473+0000","last_became_peered":"2024-03-26T21:39:47.608473+0000","last_unstale":"2024-03-26T21:40:26.725497+0000","last_undegraded":"2024-03-26T21:40:26.725497+0000","last_fullsized":"2024-03-26T21:39:47.604748+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:32.918100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:38:32.918100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:30.824532+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":444,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.739352+0000","last_change":"2024-03-26T21:39:47.634238+0000","last_active":"2024-03-26T21:40:26.739352+0000","last_peered":"2024-03-26T21:40:26.739352+0000","last_clean":"2024-03-26T21:39:37.739277+0000","last_became_active":"2024-03-26T21:39:47.634238+0000","last_became_peered":"2024-03-26T21:39:47.634238+0000","last_unstale":"2024-03-26T21:40:26.739352+0000","last_undegraded":"2024-03-26T21:40:26.739352+0000","last_fullsized":"2024-03-26T21:39:47.611412+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:38:52.919189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":464,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.745556+0000","last_change":"2024-03-26T21:39:47.625454+0000","last_active":"2024-03-26T21:40:26.745556+0000","last_peered":"2024-03-26T21:40:26.745556+0000","last_clean":"2024-03-26T21:39:37.766670+0000","last_became_active":"2024-03-26T21:39:47.625454+0000","last_became_peered":"2024-03-26T21:39:47.625454+0000","last_unstale":"2024-03-26T21:40:26.745556+0000","last_undegraded":"2024-03-26T21:39:47.609513+0000","last_fullsized":"2024-03-26T21:39:47.609410+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:29.152531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.152531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:49.669579+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":430,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.740876+0000","last_change":"2024-03-26T21:40:17.751277+0000","last_active":"2024-03-26T21:40:26.740876+0000","last_peered":"2024-03-26T21:40:26.740876+0000","last_clean":"2024-03-26T21:40:26.740876+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:40:26.740876+0000","last_undegraded":"2024-03-26T21:40:26.740876+0000","last_fullsized":"2024-03-26T21:40:26.740876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.748978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.748978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:38.847568+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.747106+0000","last_change":"2024-03-26T21:38:58.165485+0000","last_active":"2024-03-26T21:40:26.747106+0000","last_peered":"2024-03-26T21:40:26.747106+0000","last_clean":"2024-03-26T21:40:26.747106+0000","last_became_active":"2024-03-26T21:28:38.865218+0000","last_became_peered":"2024-03-26T21:28:38.865218+0000","last_unstale":"2024-03-26T21:40:26.747106+0000","last_undegraded":"2024-03-26T21:40:26.747106+0000","last_fullsized":"2024-03-26T21:40:26.747106+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:58.165359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:38:58.165359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":355,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.746577+0000","last_change":"2024-03-26T21:39:47.617946+0000","last_active":"2024-03-26T21:40:26.746577+0000","last_peered":"2024-03-26T21:40:26.746577+0000","last_clean":"2024-03-26T21:36:43.774860+0000","last_became_active":"2024-03-26T21:39:47.617946+0000","last_became_peered":"2024-03-26T21:39:47.617946+0000","last_unstale":"2024-03-26T21:40:26.746577+0000","last_undegraded":"2024-03-26T21:40:26.746577+0000","last_fullsized":"2024-03-26T21:39:47.607429+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":475,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.732874+0000","last_change":"2024-03-26T21:39:44.903744+0000","last_active":"2024-03-26T21:40:26.732874+0000","last_peered":"2024-03-26T21:40:26.732874+0000","last_clean":"2024-03-26T21:40:26.732874+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:40:26.732874+0000","last_undegraded":"2024-03-26T21:40:26.732874+0000","last_fullsized":"2024-03-26T21:40:26.732874+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:44.903549+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:39:44.903549+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:59.488494+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.745287+0000","last_change":"2024-03-26T21:38:51.032070+0000","last_active":"2024-03-26T21:40:26.745287+0000","last_peered":"2024-03-26T21:40:26.745287+0000","last_clean":"2024-03-26T21:40:26.745287+0000","last_became_active":"2024-03-26T21:28:40.891443+0000","last_became_peered":"2024-03-26T21:28:40.891443+0000","last_unstale":"2024-03-26T21:40:26.745287+0000","last_undegraded":"2024-03-26T21:40:26.745287+0000","last_fullsized":"2024-03-26T21:40:26.745287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:51.031973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:51.031973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":450,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.732655+0000","last_change":"2024-03-26T21:39:26.462178+0000","last_active":"2024-03-26T21:40:26.732655+0000","last_peered":"2024-03-26T21:40:26.732655+0000","last_clean":"2024-03-26T21:40:26.732655+0000","last_became_active":"2024-03-26T21:28:38.868295+0000","last_became_peered":"2024-03-26T21:28:38.868295+0000","last_unstale":"2024-03-26T21:40:26.732655+0000","last_undegraded":"2024-03-26T21:40:26.732655+0000","last_fullsized":"2024-03-26T21:40:26.732655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:26.461383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:26.461383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.431602+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":133,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.726080+0000","last_change":"2024-03-26T21:39:47.619518+0000","last_active":"2024-03-26T21:40:26.726080+0000","last_peered":"2024-03-26T21:40:26.726080+0000","last_clean":"2024-03-26T21:39:37.712009+0000","last_became_active":"2024-03-26T21:39:47.619518+0000","last_became_peered":"2024-03-26T21:39:47.619518+0000","last_unstale":"2024-03-26T21:40:26.726080+0000","last_undegraded":"2024-03-26T21:40:26.726080+0000","last_fullsized":"2024-03-26T21:39:47.606602+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:43.883883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:43.883883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:30.824549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":162,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726347+0000","last_change":"2024-03-26T21:39:32.823405+0000","last_active":"2024-03-26T21:40:26.726347+0000","last_peered":"2024-03-26T21:40:26.726347+0000","last_clean":"2024-03-26T21:40:26.726347+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:40:26.726347+0000","last_undegraded":"2024-03-26T21:40:26.726347+0000","last_fullsized":"2024-03-26T21:40:26.726347+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:32.823241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:32.823241+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:50.065945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":425,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.741655+0000","last_change":"2024-03-26T21:39:47.628132+0000","last_active":"2024-03-26T21:40:26.741655+0000","last_peered":"2024-03-26T21:40:26.741655+0000","last_clean":"2024-03-26T21:39:37.738458+0000","last_became_active":"2024-03-26T21:39:47.628132+0000","last_became_peered":"2024-03-26T21:39:47.628132+0000","last_unstale":"2024-03-26T21:40:26.741655+0000","last_undegraded":"2024-03-26T21:39:47.607946+0000","last_fullsized":"2024-03-26T21:39:47.607829+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:38:40.496984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.496984+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":586,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738856+0000","last_change":"2024-03-26T21:39:33.100038+0000","last_active":"2024-03-26T21:40:26.738856+0000","last_peered":"2024-03-26T21:40:26.738856+0000","last_clean":"2024-03-26T21:40:26.738856+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:40:26.738856+0000","last_undegraded":"2024-03-26T21:40:26.738856+0000","last_fullsized":"2024-03-26T21:40:26.738856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:39:33.099004+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.099004+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:51.054438+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":355,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.725996+0000","last_change":"2024-03-26T21:39:47.613318+0000","last_active":"2024-03-26T21:40:26.725996+0000","last_peered":"2024-03-26T21:40:26.725996+0000","last_clean":"2024-03-26T21:36:43.772937+0000","last_became_active":"2024-03-26T21:39:47.613318+0000","last_became_peered":"2024-03-26T21:39:47.613318+0000","last_unstale":"2024-03-26T21:40:26.725996+0000","last_undegraded":"2024-03-26T21:40:26.725996+0000","last_fullsized":"2024-03-26T21:39:47.605318+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:39.962543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:39.962543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":468,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.745260+0000","last_change":"2024-03-26T21:39:47.635305+0000","last_active":"2024-03-26T21:40:26.745260+0000","last_peered":"2024-03-26T21:40:26.745260+0000","last_clean":"2024-03-26T21:39:37.769681+0000","last_became_active":"2024-03-26T21:39:47.635305+0000","last_became_peered":"2024-03-26T21:39:47.635305+0000","last_unstale":"2024-03-26T21:40:26.745260+0000","last_undegraded":"2024-03-26T21:39:47.619459+0000","last_fullsized":"2024-03-26T21:39:47.619328+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15","last_scrub_stamp":"2024-03-26T21:37:38.528979+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:37:38.528979+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.727954+0000","last_change":"2024-03-26T21:40:19.858952+0000","last_active":"2024-03-26T21:40:26.727954+0000","last_peered":"2024-03-26T21:40:26.727954+0000","last_clean":"2024-03-26T21:40:26.727954+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:40:26.727954+0000","last_undegraded":"2024-03-26T21:40:26.727954+0000","last_fullsized":"2024-03-26T21:40:26.727954+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:19.858839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:40:19.858839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:33.273693+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"250'9","reported_seq":585,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.754051+0000","last_change":"2024-03-26T21:39:47.632090+0000","last_active":"2024-03-26T21:40:26.754051+0000","last_peered":"2024-03-26T21:40:26.754051+0000","last_clean":"2024-03-26T21:39:45.025600+0000","last_became_active":"2024-03-26T21:39:47.632090+0000","last_became_peered":"2024-03-26T21:39:47.632090+0000","last_unstale":"2024-03-26T21:40:26.754051+0000","last_undegraded":"2024-03-26T21:39:47.619918+0000","last_fullsized":"2024-03-26T21:39:47.619807+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:39:27.133215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:27.133215+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:40.158523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.741062+0000","last_change":"2024-03-26T21:39:47.627048+0000","last_active":"2024-03-26T21:40:26.741062+0000","last_peered":"2024-03-26T21:40:26.741062+0000","last_clean":"2024-03-26T21:39:37.738248+0000","last_became_active":"2024-03-26T21:39:47.627048+0000","last_became_peered":"2024-03-26T21:39:47.627048+0000","last_unstale":"2024-03-26T21:40:26.741062+0000","last_undegraded":"2024-03-26T21:40:26.741062+0000","last_fullsized":"2024-03-26T21:39:47.608878+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.378273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.378273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":469,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.739101+0000","last_change":"2024-03-26T21:39:47.620179+0000","last_active":"2024-03-26T21:40:26.739101+0000","last_peered":"2024-03-26T21:40:26.739101+0000","last_clean":"2024-03-26T21:39:37.739061+0000","last_became_active":"2024-03-26T21:39:47.620179+0000","last_became_peered":"2024-03-26T21:39:47.620179+0000","last_unstale":"2024-03-26T21:40:26.739101+0000","last_undegraded":"2024-03-26T21:39:47.610866+0000","last_fullsized":"2024-03-26T21:39:47.610753+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'13","last_scrub_stamp":"2024-03-26T21:38:33.018662+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:38:33.018662+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:30.019609+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":454,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.751669+0000","last_change":"2024-03-26T21:39:47.614459+0000","last_active":"2024-03-26T21:40:26.751669+0000","last_peered":"2024-03-26T21:40:26.751669+0000","last_clean":"2024-03-26T21:39:37.713427+0000","last_became_active":"2024-03-26T21:39:47.614459+0000","last_became_peered":"2024-03-26T21:39:47.614459+0000","last_unstale":"2024-03-26T21:40:26.751669+0000","last_undegraded":"2024-03-26T21:40:26.751669+0000","last_fullsized":"2024-03-26T21:39:47.611057+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:54.769540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:54.769540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:31.282491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":420,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.756799+0000","last_change":"2024-03-26T21:39:47.631662+0000","last_active":"2024-03-26T21:40:26.756799+0000","last_peered":"2024-03-26T21:40:26.756799+0000","last_clean":"2024-03-26T21:39:37.770374+0000","last_became_active":"2024-03-26T21:39:47.631662+0000","last_became_peered":"2024-03-26T21:39:47.631662+0000","last_unstale":"2024-03-26T21:40:26.756799+0000","last_undegraded":"2024-03-26T21:40:26.756799+0000","last_fullsized":"2024-03-26T21:39:47.618949+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:36.335509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:35:36.335509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:28.365548+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":355,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.725557+0000","last_change":"2024-03-26T21:39:47.609960+0000","last_active":"2024-03-26T21:40:26.725557+0000","last_peered":"2024-03-26T21:40:26.725557+0000","last_clean":"2024-03-26T21:36:43.773743+0000","last_became_active":"2024-03-26T21:39:47.609960+0000","last_became_peered":"2024-03-26T21:39:47.609960+0000","last_unstale":"2024-03-26T21:40:26.725557+0000","last_undegraded":"2024-03-26T21:40:26.725557+0000","last_fullsized":"2024-03-26T21:39:47.604870+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.962120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.962120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:57.319695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":461,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.742191+0000","last_change":"2024-03-26T21:39:47.628548+0000","last_active":"2024-03-26T21:40:26.742191+0000","last_peered":"2024-03-26T21:40:26.742191+0000","last_clean":"2024-03-26T21:39:37.766225+0000","last_became_active":"2024-03-26T21:39:47.628548+0000","last_became_peered":"2024-03-26T21:39:47.628548+0000","last_unstale":"2024-03-26T21:40:26.742191+0000","last_undegraded":"2024-03-26T21:39:47.611067+0000","last_fullsized":"2024-03-26T21:39:47.610933+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'13","last_scrub_stamp":"2024-03-26T21:37:51.120100+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:37:51.120100+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":343,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.733285+0000","last_change":"2024-03-26T21:39:47.618768+0000","last_active":"2024-03-26T21:40:26.733285+0000","last_peered":"2024-03-26T21:40:26.733285+0000","last_clean":"2024-03-26T21:36:43.772776+0000","last_became_active":"2024-03-26T21:39:47.618768+0000","last_became_peered":"2024-03-26T21:39:47.618768+0000","last_unstale":"2024-03-26T21:40:26.733285+0000","last_undegraded":"2024-03-26T21:40:26.733285+0000","last_fullsized":"2024-03-26T21:39:47.612327+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.068375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.739615+0000","last_change":"2024-03-26T21:38:40.964058+0000","last_active":"2024-03-26T21:40:26.739615+0000","last_peered":"2024-03-26T21:40:26.739615+0000","last_clean":"2024-03-26T21:40:26.739615+0000","last_became_active":"2024-03-26T21:28:38.874512+0000","last_became_peered":"2024-03-26T21:28:38.874512+0000","last_unstale":"2024-03-26T21:40:26.739615+0000","last_undegraded":"2024-03-26T21:40:26.739615+0000","last_fullsized":"2024-03-26T21:40:26.739615+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:40.963953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.963953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":450,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.733922+0000","last_change":"2024-03-26T21:40:19.931360+0000","last_active":"2024-03-26T21:40:26.733922+0000","last_peered":"2024-03-26T21:40:26.733922+0000","last_clean":"2024-03-26T21:40:26.733922+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:40:26.733922+0000","last_undegraded":"2024-03-26T21:40:26.733922+0000","last_fullsized":"2024-03-26T21:40:26.733922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:19.931220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:19.931220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:32.787468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":456,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.756062+0000","last_change":"2024-03-26T21:39:47.632247+0000","last_active":"2024-03-26T21:40:26.756062+0000","last_peered":"2024-03-26T21:40:26.756062+0000","last_clean":"2024-03-26T21:39:37.770213+0000","last_became_active":"2024-03-26T21:39:47.632247+0000","last_became_peered":"2024-03-26T21:39:47.632247+0000","last_unstale":"2024-03-26T21:40:26.756062+0000","last_undegraded":"2024-03-26T21:39:47.618538+0000","last_fullsized":"2024-03-26T21:39:47.618427+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-26T21:35:38.321844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:38.321844+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:28.365534+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"260'6441","reported_seq":6493,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:28.290160+0000","last_change":"2024-03-26T21:39:47.620863+0000","last_active":"2024-03-26T21:40:28.290160+0000","last_peered":"2024-03-26T21:40:28.290160+0000","last_clean":"2024-03-26T21:39:46.588667+0000","last_became_active":"2024-03-26T21:39:47.620863+0000","last_became_peered":"2024-03-26T21:39:47.620863+0000","last_unstale":"2024-03-26T21:40:28.290160+0000","last_undegraded":"2024-03-26T21:39:47.612947+0000","last_fullsized":"2024-03-26T21:39:47.612815+0000","mapping_epoch":249,"log_start":"244'400","ondisk_log_start":"244'400","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":6041,"log_dups_size":0,"ondisk_log_size":6041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422117376,"num_objects":6441,"num_object_clones":0,"num_object_copies":19323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6441,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":6441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":430,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738810+0000","last_change":"2024-03-26T21:40:25.046799+0000","last_active":"2024-03-26T21:40:26.738810+0000","last_peered":"2024-03-26T21:40:26.738810+0000","last_clean":"2024-03-26T21:40:26.738810+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:40:26.738810+0000","last_undegraded":"2024-03-26T21:40:26.738810+0000","last_fullsized":"2024-03-26T21:40:26.738810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:25.046675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.046675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:46.150005+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":373,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.727801+0000","last_change":"2024-03-26T21:39:47.616627+0000","last_active":"2024-03-26T21:40:26.727801+0000","last_peered":"2024-03-26T21:40:26.727801+0000","last_clean":"2024-03-26T21:36:43.775543+0000","last_became_active":"2024-03-26T21:39:47.616627+0000","last_became_peered":"2024-03-26T21:39:47.616627+0000","last_unstale":"2024-03-26T21:40:26.727801+0000","last_undegraded":"2024-03-26T21:39:47.607204+0000","last_fullsized":"2024-03-26T21:39:47.607093+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:39:22.057217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:22.057217+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:46.741473+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":484,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.738057+0000","last_change":"2024-03-26T21:39:47.614890+0000","last_active":"2024-03-26T21:40:26.738057+0000","last_peered":"2024-03-26T21:40:26.738057+0000","last_clean":"2024-03-26T21:39:37.736858+0000","last_became_active":"2024-03-26T21:39:47.614890+0000","last_became_peered":"2024-03-26T21:39:47.614890+0000","last_unstale":"2024-03-26T21:40:26.738057+0000","last_undegraded":"2024-03-26T21:39:47.609376+0000","last_fullsized":"2024-03-26T21:39:47.609252+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:38:59.027826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:38:59.027826+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:28.564096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"250'13","reported_seq":64,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.732769+0000","last_change":"2024-03-26T21:39:47.615404+0000","last_active":"2024-03-26T21:40:26.732769+0000","last_peered":"2024-03-26T21:40:26.732769+0000","last_clean":"2024-03-26T21:28:42.051275+0000","last_became_active":"2024-03-26T21:39:47.615404+0000","last_became_peered":"2024-03-26T21:39:47.615404+0000","last_unstale":"2024-03-26T21:40:26.732769+0000","last_undegraded":"2024-03-26T21:39:47.608113+0000","last_fullsized":"2024-03-26T21:39:47.607766+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:39:25.023169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.023169+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:40.605846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"260'6583","reported_seq":6701,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:32.292536+0000","last_change":"2024-03-26T21:40:23.214665+0000","last_active":"2024-03-26T21:40:32.292536+0000","last_peered":"2024-03-26T21:40:32.292536+0000","last_clean":"2024-03-26T21:40:32.292536+0000","last_became_active":"2024-03-26T21:39:12.241965+0000","last_became_peered":"2024-03-26T21:39:12.241965+0000","last_unstale":"2024-03-26T21:40:32.292536+0000","last_undegraded":"2024-03-26T21:40:32.292536+0000","last_fullsized":"2024-03-26T21:40:32.292536+0000","mapping_epoch":240,"log_start":"245'1800","ondisk_log_start":"245'1800","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'5979","last_scrub_stamp":"2024-03-26T21:40:23.214462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.214462+0000","objects_scrubbed":5893,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:35.154863+0000","scrub_duration":2352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431423488,"num_objects":6583,"num_object_clones":0,"num_object_copies":19749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6583,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":417,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.747203+0000","last_change":"2024-03-26T21:39:47.632525+0000","last_active":"2024-03-26T21:40:26.747203+0000","last_peered":"2024-03-26T21:40:26.747203+0000","last_clean":"2024-03-26T21:39:37.770596+0000","last_became_active":"2024-03-26T21:39:47.632525+0000","last_became_peered":"2024-03-26T21:39:47.632525+0000","last_unstale":"2024-03-26T21:40:26.747203+0000","last_undegraded":"2024-03-26T21:40:26.747203+0000","last_fullsized":"2024-03-26T21:39:47.616460+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:35:45.377075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:35:45.377075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:28.365561+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":366,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.727636+0000","last_change":"2024-03-26T21:39:47.630843+0000","last_active":"2024-03-26T21:40:26.727636+0000","last_peered":"2024-03-26T21:40:26.727636+0000","last_clean":"2024-03-26T21:36:43.775402+0000","last_became_active":"2024-03-26T21:39:47.630843+0000","last_became_peered":"2024-03-26T21:39:47.630843+0000","last_unstale":"2024-03-26T21:40:26.727636+0000","last_undegraded":"2024-03-26T21:39:47.606971+0000","last_fullsized":"2024-03-26T21:39:47.606836+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:39:32.049629+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:39:32.049629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:46.737296+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"234'18","reported_seq":151,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.725944+0000","last_change":"2024-03-26T21:39:31.801649+0000","last_active":"2024-03-26T21:40:26.725944+0000","last_peered":"2024-03-26T21:40:26.725944+0000","last_clean":"2024-03-26T21:40:26.725944+0000","last_became_active":"2024-03-26T21:36:51.690121+0000","last_became_peered":"2024-03-26T21:36:51.690121+0000","last_unstale":"2024-03-26T21:40:26.725944+0000","last_undegraded":"2024-03-26T21:40:26.725944+0000","last_fullsized":"2024-03-26T21:40:26.725944+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-26T21:39:31.801539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:31.801539+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:51.347650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.756311+0000","last_change":"2024-03-26T21:39:24.029907+0000","last_active":"2024-03-26T21:40:26.756311+0000","last_peered":"2024-03-26T21:40:26.756311+0000","last_clean":"2024-03-26T21:40:26.756311+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:40:26.756311+0000","last_undegraded":"2024-03-26T21:40:26.756311+0000","last_fullsized":"2024-03-26T21:40:26.756311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:24.029833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.029833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:30.867251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"260'6618","reported_seq":6672,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:31.623786+0000","last_change":"2024-03-26T21:39:47.655178+0000","last_active":"2024-03-26T21:40:31.623786+0000","last_peered":"2024-03-26T21:40:31.623786+0000","last_clean":"2024-03-26T21:39:46.590740+0000","last_became_active":"2024-03-26T21:39:47.637407+0000","last_became_peered":"2024-03-26T21:39:47.637407+0000","last_unstale":"2024-03-26T21:40:31.623786+0000","last_undegraded":"2024-03-26T21:39:47.618198+0000","last_fullsized":"2024-03-26T21:39:47.611586+0000","mapping_epoch":249,"log_start":"244'1400","ondisk_log_start":"244'1400","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433717248,"num_objects":6618,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6618,"num_write_kb":423552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":6618}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":29,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.755990+0000","last_change":"2024-03-26T21:39:47.631297+0000","last_active":"2024-03-26T21:40:26.755990+0000","last_peered":"2024-03-26T21:40:26.755990+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:39:47.631297+0000","last_became_peered":"2024-03-26T21:39:47.631297+0000","last_unstale":"2024-03-26T21:40:26.755990+0000","last_undegraded":"2024-03-26T21:40:26.755990+0000","last_fullsized":"2024-03-26T21:39:47.612796+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:57.028707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:57.028707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:31.282498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":450,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.751041+0000","last_change":"2024-03-26T21:39:19.956045+0000","last_active":"2024-03-26T21:40:26.751041+0000","last_peered":"2024-03-26T21:40:26.751041+0000","last_clean":"2024-03-26T21:40:26.751041+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:40:26.751041+0000","last_undegraded":"2024-03-26T21:40:26.751041+0000","last_fullsized":"2024-03-26T21:40:26.751041+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:19.955968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:19.955968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.127249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.738967+0000","last_change":"2024-03-26T21:39:47.618149+0000","last_active":"2024-03-26T21:40:26.738967+0000","last_peered":"2024-03-26T21:40:26.738967+0000","last_clean":"2024-03-26T21:28:33.816134+0000","last_became_active":"2024-03-26T21:39:47.618149+0000","last_became_peered":"2024-03-26T21:39:47.618149+0000","last_unstale":"2024-03-26T21:40:26.738967+0000","last_undegraded":"2024-03-26T21:40:26.738967+0000","last_fullsized":"2024-03-26T21:39:47.610702+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.044612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.044612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:47.241682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":513,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.732784+0000","last_change":"2024-03-26T21:39:47.632979+0000","last_active":"2024-03-26T21:40:26.732784+0000","last_peered":"2024-03-26T21:40:26.732784+0000","last_clean":"2024-03-26T21:39:37.768179+0000","last_became_active":"2024-03-26T21:39:47.632979+0000","last_became_peered":"2024-03-26T21:39:47.632979+0000","last_unstale":"2024-03-26T21:40:26.732784+0000","last_undegraded":"2024-03-26T21:39:47.614795+0000","last_fullsized":"2024-03-26T21:39:47.614676+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:38:51.982227+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:38:51.982227+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"260'6841","reported_seq":6891,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:31.629856+0000","last_change":"2024-03-26T21:39:12.242167+0000","last_active":"2024-03-26T21:40:31.629856+0000","last_peered":"2024-03-26T21:40:31.629856+0000","last_clean":"2024-03-26T21:40:31.629856+0000","last_became_active":"2024-03-26T21:39:12.241832+0000","last_became_peered":"2024-03-26T21:39:12.241832+0000","last_unstale":"2024-03-26T21:40:31.629856+0000","last_undegraded":"2024-03-26T21:40:31.629856+0000","last_fullsized":"2024-03-26T21:40:31.629856+0000","mapping_epoch":240,"log_start":"244'1600","ondisk_log_start":"244'1600","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448331776,"num_objects":6841,"num_object_clones":0,"num_object_copies":20523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6841,"num_write_kb":437824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.747474+0000","last_change":"2024-03-26T21:39:47.614378+0000","last_active":"2024-03-26T21:40:26.747474+0000","last_peered":"2024-03-26T21:40:26.747474+0000","last_clean":"2024-03-26T21:39:38.253970+0000","last_became_active":"2024-03-26T21:39:47.614378+0000","last_became_peered":"2024-03-26T21:39:47.614378+0000","last_unstale":"2024-03-26T21:40:26.747474+0000","last_undegraded":"2024-03-26T21:40:26.747474+0000","last_fullsized":"2024-03-26T21:39:47.608171+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:41.522724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:38:41.522724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":463,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.751501+0000","last_change":"2024-03-26T21:40:23.255141+0000","last_active":"2024-03-26T21:40:26.751501+0000","last_peered":"2024-03-26T21:40:26.751501+0000","last_clean":"2024-03-26T21:40:26.751501+0000","last_became_active":"2024-03-26T21:36:51.707172+0000","last_became_peered":"2024-03-26T21:36:51.707172+0000","last_unstale":"2024-03-26T21:40:26.751501+0000","last_undegraded":"2024-03-26T21:40:26.751501+0000","last_fullsized":"2024-03-26T21:40:26.751501+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:23.254687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:46.735203+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.254687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:42.677450+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":450,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738563+0000","last_change":"2024-03-26T21:40:13.989757+0000","last_active":"2024-03-26T21:40:26.738563+0000","last_peered":"2024-03-26T21:40:26.738563+0000","last_clean":"2024-03-26T21:40:26.738563+0000","last_became_active":"2024-03-26T21:28:34.841391+0000","last_became_peered":"2024-03-26T21:28:34.841391+0000","last_unstale":"2024-03-26T21:40:26.738563+0000","last_undegraded":"2024-03-26T21:40:26.738563+0000","last_fullsized":"2024-03-26T21:40:26.738563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:13.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:40:13.989637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:30.618190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":488,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.743592+0000","last_change":"2024-03-26T21:39:46.369958+0000","last_active":"2024-03-26T21:40:26.743592+0000","last_peered":"2024-03-26T21:40:26.743592+0000","last_clean":"2024-03-26T21:40:26.743592+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:40:26.743592+0000","last_undegraded":"2024-03-26T21:40:26.743592+0000","last_fullsized":"2024-03-26T21:40:26.743592+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:39:46.369461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:46.369461+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:51.145247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"260'6451","reported_seq":6504,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:28.299634+0000","last_change":"2024-03-26T21:39:47.613928+0000","last_active":"2024-03-26T21:40:28.299634+0000","last_peered":"2024-03-26T21:40:28.299634+0000","last_clean":"2024-03-26T21:39:46.592847+0000","last_became_active":"2024-03-26T21:39:47.613928+0000","last_became_peered":"2024-03-26T21:39:47.613928+0000","last_unstale":"2024-03-26T21:40:28.299634+0000","last_undegraded":"2024-03-26T21:39:47.607620+0000","last_fullsized":"2024-03-26T21:39:47.607497+0000","mapping_epoch":249,"log_start":"244'400","ondisk_log_start":"244'400","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":6051,"log_dups_size":0,"ondisk_log_size":6051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:31.036053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422772736,"num_objects":6451,"num_object_clones":0,"num_object_copies":19353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6451,"num_write_kb":412864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":6451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":438,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.743757+0000","last_change":"2024-03-26T21:39:47.613382+0000","last_active":"2024-03-26T21:40:26.743757+0000","last_peered":"2024-03-26T21:40:26.743757+0000","last_clean":"2024-03-26T21:39:37.769345+0000","last_became_active":"2024-03-26T21:39:47.613382+0000","last_became_peered":"2024-03-26T21:39:47.613382+0000","last_unstale":"2024-03-26T21:40:26.743757+0000","last_undegraded":"2024-03-26T21:40:26.743757+0000","last_fullsized":"2024-03-26T21:39:47.608036+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:31.533053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:37:31.533053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":452,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738542+0000","last_change":"2024-03-26T21:38:57.985359+0000","last_active":"2024-03-26T21:40:26.738542+0000","last_peered":"2024-03-26T21:40:26.738542+0000","last_clean":"2024-03-26T21:40:26.738542+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:40:26.738542+0000","last_undegraded":"2024-03-26T21:40:26.738542+0000","last_fullsized":"2024-03-26T21:40:26.738542+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_clean_scrub_stamp":"2024-03-26T21:38:57.985228+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726248+0000","last_change":"2024-03-26T21:39:35.731965+0000","last_active":"2024-03-26T21:40:26.726248+0000","last_peered":"2024-03-26T21:40:26.726248+0000","last_clean":"2024-03-26T21:40:26.726248+0000","last_became_active":"2024-03-26T21:36:51.695776+0000","last_became_peered":"2024-03-26T21:36:51.695776+0000","last_unstale":"2024-03-26T21:40:26.726248+0000","last_undegraded":"2024-03-26T21:40:26.726248+0000","last_fullsized":"2024-03-26T21:40:26.726248+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:35.731888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:35.731888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:56.484214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"234'41","reported_seq":602,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738120+0000","last_change":"2024-03-26T21:39:19.161551+0000","last_active":"2024-03-26T21:40:26.738120+0000","last_peered":"2024-03-26T21:40:26.738120+0000","last_clean":"2024-03-26T21:40:26.738120+0000","last_became_active":"2024-03-26T21:28:59.678271+0000","last_became_peered":"2024-03-26T21:28:59.678271+0000","last_unstale":"2024-03-26T21:40:26.738120+0000","last_undegraded":"2024-03-26T21:40:26.738120+0000","last_fullsized":"2024-03-26T21:40:26.738120+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'41","last_scrub_stamp":"2024-03-26T21:39:19.161453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:19.161453+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:44.911931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.745720+0000","last_change":"2024-03-26T21:39:34.329554+0000","last_active":"2024-03-26T21:40:26.745720+0000","last_peered":"2024-03-26T21:40:26.745720+0000","last_clean":"2024-03-26T21:40:26.745720+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:40:26.745720+0000","last_undegraded":"2024-03-26T21:40:26.745720+0000","last_fullsized":"2024-03-26T21:40:26.745720+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:34.329433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:34.329433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:36.095486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"260'6676","reported_seq":6728,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:31.591735+0000","last_change":"2024-03-26T21:39:47.634284+0000","last_active":"2024-03-26T21:40:31.591735+0000","last_peered":"2024-03-26T21:40:31.591735+0000","last_clean":"2024-03-26T21:39:46.599316+0000","last_became_active":"2024-03-26T21:39:47.634284+0000","last_became_peered":"2024-03-26T21:39:47.634284+0000","last_unstale":"2024-03-26T21:40:31.591735+0000","last_undegraded":"2024-03-26T21:39:47.609322+0000","last_fullsized":"2024-03-26T21:39:47.609190+0000","mapping_epoch":249,"log_start":"244'1500","ondisk_log_start":"244'1500","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437518336,"num_objects":6676,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6676,"num_write_kb":427264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6676}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":29,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.733147+0000","last_change":"2024-03-26T21:39:47.631284+0000","last_active":"2024-03-26T21:40:26.733147+0000","last_peered":"2024-03-26T21:40:26.733147+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:39:47.631284+0000","last_became_peered":"2024-03-26T21:39:47.631284+0000","last_unstale":"2024-03-26T21:40:26.733147+0000","last_undegraded":"2024-03-26T21:40:26.733147+0000","last_fullsized":"2024-03-26T21:39:47.615703+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:44.031272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:38:44.031272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:27.990190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":458,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.734476+0000","last_change":"2024-03-26T21:39:38.833455+0000","last_active":"2024-03-26T21:40:26.734476+0000","last_peered":"2024-03-26T21:40:26.734476+0000","last_clean":"2024-03-26T21:40:26.734476+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:40:26.734476+0000","last_undegraded":"2024-03-26T21:40:26.734476+0000","last_fullsized":"2024-03-26T21:40:26.734476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:38.833257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:39:38.833257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:44.790767+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":457,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.750812+0000","last_change":"2024-03-26T21:40:19.294625+0000","last_active":"2024-03-26T21:40:26.750812+0000","last_peered":"2024-03-26T21:40:26.750812+0000","last_clean":"2024-03-26T21:40:26.750812+0000","last_became_active":"2024-03-26T21:28:34.844965+0000","last_became_peered":"2024-03-26T21:28:34.844965+0000","last_unstale":"2024-03-26T21:40:26.750812+0000","last_undegraded":"2024-03-26T21:40:26.750812+0000","last_fullsized":"2024-03-26T21:40:26.750812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:19.294462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:40:19.294462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:28.617547+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":492,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.736391+0000","last_change":"2024-03-26T21:40:17.856788+0000","last_active":"2024-03-26T21:40:26.736391+0000","last_peered":"2024-03-26T21:40:26.736391+0000","last_clean":"2024-03-26T21:40:26.736391+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:40:26.736391+0000","last_undegraded":"2024-03-26T21:40:26.736391+0000","last_fullsized":"2024-03-26T21:40:26.736391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:40:17.856643+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.856643+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:27.513445+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.751220+0000","last_change":"2024-03-26T21:39:47.620668+0000","last_active":"2024-03-26T21:40:26.751220+0000","last_peered":"2024-03-26T21:40:26.751220+0000","last_clean":"2024-03-26T21:39:37.733196+0000","last_became_active":"2024-03-26T21:39:47.620668+0000","last_became_peered":"2024-03-26T21:39:47.620668+0000","last_unstale":"2024-03-26T21:40:26.751220+0000","last_undegraded":"2024-03-26T21:40:26.751220+0000","last_fullsized":"2024-03-26T21:39:47.612459+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:31.766377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:38:31.766377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:30.240815+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"260'6779","reported_seq":6829,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:32.503207+0000","last_change":"2024-03-26T21:39:12.147845+0000","last_active":"2024-03-26T21:40:32.503207+0000","last_peered":"2024-03-26T21:40:32.503207+0000","last_clean":"2024-03-26T21:40:32.503207+0000","last_became_active":"2024-03-26T21:39:12.147552+0000","last_became_peered":"2024-03-26T21:39:12.147552+0000","last_unstale":"2024-03-26T21:40:32.503207+0000","last_undegraded":"2024-03-26T21:40:32.503207+0000","last_fullsized":"2024-03-26T21:40:32.503207+0000","mapping_epoch":240,"log_start":"246'2000","ondisk_log_start":"246'2000","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:35.535409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444268544,"num_objects":6779,"num_object_clones":0,"num_object_copies":20337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":443,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.739791+0000","last_change":"2024-03-26T21:38:49.921902+0000","last_active":"2024-03-26T21:40:26.739791+0000","last_peered":"2024-03-26T21:40:26.739791+0000","last_clean":"2024-03-26T21:40:26.739791+0000","last_became_active":"2024-03-26T21:36:51.678088+0000","last_became_peered":"2024-03-26T21:36:51.678088+0000","last_unstale":"2024-03-26T21:40:26.739791+0000","last_undegraded":"2024-03-26T21:40:26.739791+0000","last_fullsized":"2024-03-26T21:40:26.739791+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:49.921786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:49.921786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":321,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.741185+0000","last_change":"2024-03-26T21:39:47.620024+0000","last_active":"2024-03-26T21:40:26.741185+0000","last_peered":"2024-03-26T21:40:26.741185+0000","last_clean":"2024-03-26T21:29:17.368539+0000","last_became_active":"2024-03-26T21:39:47.620024+0000","last_became_peered":"2024-03-26T21:39:47.620024+0000","last_unstale":"2024-03-26T21:40:26.741185+0000","last_undegraded":"2024-03-26T21:39:47.612008+0000","last_fullsized":"2024-03-26T21:39:47.611890+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:39:24.057586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.057586+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.623099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.744356+0000","last_change":"2024-03-26T21:37:42.570702+0000","last_active":"2024-03-26T21:40:26.744356+0000","last_peered":"2024-03-26T21:40:26.744356+0000","last_clean":"2024-03-26T21:40:26.744356+0000","last_became_active":"2024-03-26T21:28:34.846136+0000","last_became_peered":"2024-03-26T21:28:34.846136+0000","last_unstale":"2024-03-26T21:40:26.744356+0000","last_undegraded":"2024-03-26T21:40:26.744356+0000","last_fullsized":"2024-03-26T21:40:26.744356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:42.570535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:42.570535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"250'41","reported_seq":694,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.733043+0000","last_change":"2024-03-26T21:39:47.619666+0000","last_active":"2024-03-26T21:40:26.733043+0000","last_peered":"2024-03-26T21:40:26.733043+0000","last_clean":"2024-03-26T21:39:45.023720+0000","last_became_active":"2024-03-26T21:39:47.619666+0000","last_became_peered":"2024-03-26T21:39:47.619666+0000","last_unstale":"2024-03-26T21:40:26.733043+0000","last_undegraded":"2024-03-26T21:39:47.612927+0000","last_fullsized":"2024-03-26T21:39:47.612456+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'40","last_scrub_stamp":"2024-03-26T21:38:58.044348+0000","last_deep_scrub":"88'24","last_deep_scrub_stamp":"2024-03-26T21:29:55.903057+0000","last_clean_scrub_stamp":"2024-03-26T21:38:58.044348+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.011291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.751754+0000","last_change":"2024-03-26T21:39:47.618753+0000","last_active":"2024-03-26T21:40:26.751754+0000","last_peered":"2024-03-26T21:40:26.751754+0000","last_clean":"2024-03-26T21:28:37.828962+0000","last_became_active":"2024-03-26T21:39:47.618753+0000","last_became_peered":"2024-03-26T21:39:47.618753+0000","last_unstale":"2024-03-26T21:40:26.751754+0000","last_undegraded":"2024-03-26T21:40:26.751754+0000","last_fullsized":"2024-03-26T21:39:47.612244+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:25.982541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.982541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:47.348214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"260'6352","reported_seq":6402,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:28.286952+0000","last_change":"2024-03-26T21:39:12.163334+0000","last_active":"2024-03-26T21:40:28.286952+0000","last_peered":"2024-03-26T21:40:28.286952+0000","last_clean":"2024-03-26T21:40:28.286952+0000","last_became_active":"2024-03-26T21:39:12.162957+0000","last_became_peered":"2024-03-26T21:39:12.162957+0000","last_unstale":"2024-03-26T21:40:28.286952+0000","last_undegraded":"2024-03-26T21:40:28.286952+0000","last_fullsized":"2024-03-26T21:40:28.286952+0000","mapping_epoch":240,"log_start":"244'300","ondisk_log_start":"244'300","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416284672,"num_objects":6352,"num_object_clones":0,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":406528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":446,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.731730+0000","last_change":"2024-03-26T21:40:21.926819+0000","last_active":"2024-03-26T21:40:26.731730+0000","last_peered":"2024-03-26T21:40:26.731730+0000","last_clean":"2024-03-26T21:40:26.731730+0000","last_became_active":"2024-03-26T21:28:40.887124+0000","last_became_peered":"2024-03-26T21:28:40.887124+0000","last_unstale":"2024-03-26T21:40:26.731730+0000","last_undegraded":"2024-03-26T21:40:26.731730+0000","last_fullsized":"2024-03-26T21:40:26.731730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:21.924941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:21.924941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:23.982098+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":455,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.747277+0000","last_change":"2024-03-26T21:39:42.353261+0000","last_active":"2024-03-26T21:40:26.747277+0000","last_peered":"2024-03-26T21:40:26.747277+0000","last_clean":"2024-03-26T21:40:26.747277+0000","last_became_active":"2024-03-26T21:36:51.685071+0000","last_became_peered":"2024-03-26T21:36:51.685071+0000","last_unstale":"2024-03-26T21:40:26.747277+0000","last_undegraded":"2024-03-26T21:40:26.747277+0000","last_fullsized":"2024-03-26T21:40:26.747277+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.353121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:17.708355+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.353121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:46.847899+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1150,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:30.755408+0000","last_change":"2024-03-26T21:39:47.631751+0000","last_active":"2024-03-26T21:40:30.755408+0000","last_peered":"2024-03-26T21:40:30.755408+0000","last_clean":"2024-03-26T21:39:33.058561+0000","last_became_active":"2024-03-26T21:39:47.631751+0000","last_became_peered":"2024-03-26T21:39:47.631751+0000","last_unstale":"2024-03-26T21:40:30.755408+0000","last_undegraded":"2024-03-26T21:39:47.616507+0000","last_fullsized":"2024-03-26T21:39:47.616394+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.064714+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:59.933222+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":695,"num_read_kb":690,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":494,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.741868+0000","last_change":"2024-03-26T21:40:16.682329+0000","last_active":"2024-03-26T21:40:26.741868+0000","last_peered":"2024-03-26T21:40:26.741868+0000","last_clean":"2024-03-26T21:40:26.741868+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:40:26.741868+0000","last_undegraded":"2024-03-26T21:40:26.741868+0000","last_fullsized":"2024-03-26T21:40:26.741868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:16.682154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.682154+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:35.209959+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.740034+0000","last_change":"2024-03-26T21:39:23.219325+0000","last_active":"2024-03-26T21:40:26.740034+0000","last_peered":"2024-03-26T21:40:26.740034+0000","last_clean":"2024-03-26T21:40:26.740034+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:40:26.740034+0000","last_undegraded":"2024-03-26T21:40:26.740034+0000","last_fullsized":"2024-03-26T21:40:26.740034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.219236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.219236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:29.544352+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"260'6338","reported_seq":6395,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:28.581064+0000","last_change":"2024-03-26T21:39:47.640444+0000","last_active":"2024-03-26T21:40:28.581064+0000","last_peered":"2024-03-26T21:40:28.581064+0000","last_clean":"2024-03-26T21:39:46.584606+0000","last_became_active":"2024-03-26T21:39:47.631397+0000","last_became_peered":"2024-03-26T21:39:47.631397+0000","last_unstale":"2024-03-26T21:40:28.581064+0000","last_undegraded":"2024-03-26T21:39:47.609029+0000","last_fullsized":"2024-03-26T21:39:47.607963+0000","mapping_epoch":249,"log_start":"244'1400","ondisk_log_start":"244'1400","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.512527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415367168,"num_objects":6338,"num_object_clones":0,"num_object_copies":19014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":405632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6338}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":438,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.735590+0000","last_change":"2024-03-26T21:39:39.919957+0000","last_active":"2024-03-26T21:40:26.735590+0000","last_peered":"2024-03-26T21:40:26.735590+0000","last_clean":"2024-03-26T21:40:26.735590+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:40:26.735590+0000","last_undegraded":"2024-03-26T21:40:26.735590+0000","last_fullsized":"2024-03-26T21:40:26.735590+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.919791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.919791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:02.666245+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":458,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.731943+0000","last_change":"2024-03-26T21:40:18.905838+0000","last_active":"2024-03-26T21:40:26.731943+0000","last_peered":"2024-03-26T21:40:26.731943+0000","last_clean":"2024-03-26T21:40:26.731943+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:40:26.731943+0000","last_undegraded":"2024-03-26T21:40:26.731943+0000","last_fullsized":"2024-03-26T21:40:26.731943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:18.905727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:40:18.905727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:27.232365+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":484,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.741751+0000","last_change":"2024-03-26T21:39:47.620005+0000","last_active":"2024-03-26T21:40:26.741751+0000","last_peered":"2024-03-26T21:40:26.741751+0000","last_clean":"2024-03-26T21:39:37.766871+0000","last_became_active":"2024-03-26T21:39:47.620005+0000","last_became_peered":"2024-03-26T21:39:47.620005+0000","last_unstale":"2024-03-26T21:40:26.741751+0000","last_undegraded":"2024-03-26T21:39:47.610491+0000","last_fullsized":"2024-03-26T21:39:47.610380+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:38:44.485803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:38:44.485803+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.732696+0000","last_change":"2024-03-26T21:40:22.992410+0000","last_active":"2024-03-26T21:40:26.732696+0000","last_peered":"2024-03-26T21:40:26.732696+0000","last_clean":"2024-03-26T21:40:26.732696+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:40:26.732696+0000","last_undegraded":"2024-03-26T21:40:26.732696+0000","last_fullsized":"2024-03-26T21:40:26.732696+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.992328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.992328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:42.526506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"260'6623","reported_seq":6733,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:31.612991+0000","last_change":"2024-03-26T21:40:16.575888+0000","last_active":"2024-03-26T21:40:31.612991+0000","last_peered":"2024-03-26T21:40:31.612991+0000","last_clean":"2024-03-26T21:40:31.612991+0000","last_became_active":"2024-03-26T21:39:12.145693+0000","last_became_peered":"2024-03-26T21:39:12.145693+0000","last_unstale":"2024-03-26T21:40:31.612991+0000","last_undegraded":"2024-03-26T21:40:31.612991+0000","last_fullsized":"2024-03-26T21:40:31.612991+0000","mapping_epoch":240,"log_start":"244'1400","ondisk_log_start":"244'1400","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'5548","last_scrub_stamp":"2024-03-26T21:40:16.575654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.575654+0000","objects_scrubbed":5473,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:45.966368+0000","scrub_duration":1623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434044928,"num_objects":6623,"num_object_clones":0,"num_object_copies":19869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.740997+0000","last_change":"2024-03-26T21:38:48.462706+0000","last_active":"2024-03-26T21:40:26.740997+0000","last_peered":"2024-03-26T21:40:26.740997+0000","last_clean":"2024-03-26T21:40:26.740997+0000","last_became_active":"2024-03-26T21:28:40.890466+0000","last_became_peered":"2024-03-26T21:28:40.890466+0000","last_unstale":"2024-03-26T21:40:26.740997+0000","last_undegraded":"2024-03-26T21:40:26.740997+0000","last_fullsized":"2024-03-26T21:40:26.740997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:48.462602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:48.462602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738907+0000","last_change":"2024-03-26T21:38:40.017258+0000","last_active":"2024-03-26T21:40:26.738907+0000","last_peered":"2024-03-26T21:40:26.738907+0000","last_clean":"2024-03-26T21:40:26.738907+0000","last_became_active":"2024-03-26T21:28:34.840910+0000","last_became_peered":"2024-03-26T21:28:34.840910+0000","last_unstale":"2024-03-26T21:40:26.738907+0000","last_undegraded":"2024-03-26T21:40:26.738907+0000","last_fullsized":"2024-03-26T21:40:26.738907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:40.017014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.017014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":521,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:29.967948+0000","last_change":"2024-03-26T21:40:29.968002+0000","last_active":"2024-03-26T21:40:29.967948+0000","last_peered":"2024-03-26T21:40:29.967948+0000","last_clean":"2024-03-26T21:40:29.967948+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:40:29.967948+0000","last_undegraded":"2024-03-26T21:40:29.967948+0000","last_fullsized":"2024-03-26T21:40:29.967948+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:40:29.967581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.967581+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:57.155065+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":585,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:27.726606+0000","last_change":"2024-03-26T21:40:27.726654+0000","last_active":"2024-03-26T21:40:27.726606+0000","last_peered":"2024-03-26T21:40:27.726606+0000","last_clean":"2024-03-26T21:40:27.726606+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:40:27.726606+0000","last_undegraded":"2024-03-26T21:40:27.726606+0000","last_fullsized":"2024-03-26T21:40:27.726606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:40:27.726492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:27.726492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:45.900706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"260'6632","reported_seq":6685,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:32.505283+0000","last_change":"2024-03-26T21:39:47.622247+0000","last_active":"2024-03-26T21:40:32.505283+0000","last_peered":"2024-03-26T21:40:32.505283+0000","last_clean":"2024-03-26T21:39:46.600417+0000","last_became_active":"2024-03-26T21:39:47.622247+0000","last_became_peered":"2024-03-26T21:39:47.622247+0000","last_unstale":"2024-03-26T21:40:32.505283+0000","last_undegraded":"2024-03-26T21:39:47.617271+0000","last_fullsized":"2024-03-26T21:39:47.616947+0000","mapping_epoch":249,"log_start":"246'1900","ondisk_log_start":"246'1900","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434634752,"num_objects":6632,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6632,"num_write_kb":424448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":29,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.733278+0000","last_change":"2024-03-26T21:39:47.623085+0000","last_active":"2024-03-26T21:40:26.733278+0000","last_peered":"2024-03-26T21:40:26.733278+0000","last_clean":"2024-03-26T21:28:39.845082+0000","last_became_active":"2024-03-26T21:39:47.623085+0000","last_became_peered":"2024-03-26T21:39:47.623085+0000","last_unstale":"2024-03-26T21:40:26.733278+0000","last_undegraded":"2024-03-26T21:40:26.733278+0000","last_fullsized":"2024-03-26T21:39:47.611094+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:37.982321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:37.982321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:27.990220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":456,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.741968+0000","last_change":"2024-03-26T21:39:45.352141+0000","last_active":"2024-03-26T21:40:26.741968+0000","last_peered":"2024-03-26T21:40:26.741968+0000","last_clean":"2024-03-26T21:40:26.741968+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:40:26.741968+0000","last_undegraded":"2024-03-26T21:40:26.741968+0000","last_fullsized":"2024-03-26T21:40:26.741968+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:39:45.352017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.352017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:00.712774+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":175,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.725869+0000","last_change":"2024-03-26T21:40:17.802927+0000","last_active":"2024-03-26T21:40:26.725869+0000","last_peered":"2024-03-26T21:40:26.725869+0000","last_clean":"2024-03-26T21:40:26.725869+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:40:26.725869+0000","last_undegraded":"2024-03-26T21:40:26.725869+0000","last_fullsized":"2024-03-26T21:40:26.725869+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:40:17.802828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.802828+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:31.695342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"205'12","reported_seq":610,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.743303+0000","last_change":"2024-03-26T21:40:23.374179+0000","last_active":"2024-03-26T21:40:26.743303+0000","last_peered":"2024-03-26T21:40:26.743303+0000","last_clean":"2024-03-26T21:40:26.743303+0000","last_became_active":"2024-03-26T21:36:51.690572+0000","last_became_peered":"2024-03-26T21:36:51.690572+0000","last_unstale":"2024-03-26T21:40:26.743303+0000","last_undegraded":"2024-03-26T21:40:26.743303+0000","last_fullsized":"2024-03-26T21:40:26.743303+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:40:23.374040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.374040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:42.395612+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"260'6726","reported_seq":6783,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:32.301966+0000","last_change":"2024-03-26T21:39:47.653586+0000","last_active":"2024-03-26T21:40:32.301966+0000","last_peered":"2024-03-26T21:40:32.301966+0000","last_clean":"2024-03-26T21:39:46.594125+0000","last_became_active":"2024-03-26T21:39:47.645580+0000","last_became_peered":"2024-03-26T21:39:47.645580+0000","last_unstale":"2024-03-26T21:40:32.301966+0000","last_undegraded":"2024-03-26T21:39:47.631340+0000","last_fullsized":"2024-03-26T21:39:47.617917+0000","mapping_epoch":249,"log_start":"245'1900","ondisk_log_start":"245'1900","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:34.962825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440795136,"num_objects":6726,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6726,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6726,"num_write_kb":430464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6726}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726194+0000","last_change":"2024-03-26T21:39:41.855530+0000","last_active":"2024-03-26T21:40:26.726194+0000","last_peered":"2024-03-26T21:40:26.726194+0000","last_clean":"2024-03-26T21:40:26.726194+0000","last_became_active":"2024-03-26T21:36:51.696247+0000","last_became_peered":"2024-03-26T21:36:51.696247+0000","last_unstale":"2024-03-26T21:40:26.726194+0000","last_undegraded":"2024-03-26T21:40:26.726194+0000","last_fullsized":"2024-03-26T21:40:26.726194+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:41.855418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:39:41.855418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:00.664898+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.755757+0000","last_change":"2024-03-26T21:39:47.631325+0000","last_active":"2024-03-26T21:40:26.755757+0000","last_peered":"2024-03-26T21:40:26.755757+0000","last_clean":"2024-03-26T21:39:37.770039+0000","last_became_active":"2024-03-26T21:39:47.631325+0000","last_became_peered":"2024-03-26T21:39:47.631325+0000","last_unstale":"2024-03-26T21:40:26.755757+0000","last_undegraded":"2024-03-26T21:40:26.755757+0000","last_fullsized":"2024-03-26T21:39:47.618042+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:53.052940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:53.052940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"234'30","reported_seq":475,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.743055+0000","last_change":"2024-03-26T21:39:47.616880+0000","last_active":"2024-03-26T21:40:26.743055+0000","last_peered":"2024-03-26T21:40:26.743055+0000","last_clean":"2024-03-26T21:39:37.767521+0000","last_became_active":"2024-03-26T21:39:47.616880+0000","last_became_peered":"2024-03-26T21:39:47.616880+0000","last_unstale":"2024-03-26T21:40:26.743055+0000","last_undegraded":"2024-03-26T21:39:47.608769+0000","last_fullsized":"2024-03-26T21:39:47.608658+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-26T21:35:33.264627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:35:33.264627+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:28.365514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":133,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.726058+0000","last_change":"2024-03-26T21:39:47.613145+0000","last_active":"2024-03-26T21:40:26.726058+0000","last_peered":"2024-03-26T21:40:26.726058+0000","last_clean":"2024-03-26T21:39:37.711445+0000","last_became_active":"2024-03-26T21:39:47.613145+0000","last_became_peered":"2024-03-26T21:39:47.613145+0000","last_unstale":"2024-03-26T21:40:26.726058+0000","last_undegraded":"2024-03-26T21:40:26.726058+0000","last_fullsized":"2024-03-26T21:39:47.605371+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:38:52.872296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"260'6753","reported_seq":6801,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:32.477075+0000","last_change":"2024-03-26T21:39:12.242104+0000","last_active":"2024-03-26T21:40:32.477075+0000","last_peered":"2024-03-26T21:40:32.477075+0000","last_clean":"2024-03-26T21:40:32.477075+0000","last_became_active":"2024-03-26T21:39:12.241774+0000","last_became_peered":"2024-03-26T21:39:12.241774+0000","last_unstale":"2024-03-26T21:40:32.477075+0000","last_undegraded":"2024-03-26T21:40:32.477075+0000","last_fullsized":"2024-03-26T21:40:32.477075+0000","mapping_epoch":240,"log_start":"245'2000","ondisk_log_start":"245'2000","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442564608,"num_objects":6753,"num_object_clones":0,"num_object_copies":20259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6753,"num_write_kb":432192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":430,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.738622+0000","last_change":"2024-03-26T21:39:31.189014+0000","last_active":"2024-03-26T21:40:26.738622+0000","last_peered":"2024-03-26T21:40:26.738622+0000","last_clean":"2024-03-26T21:40:26.738622+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:40:26.738622+0000","last_undegraded":"2024-03-26T21:40:26.738622+0000","last_fullsized":"2024-03-26T21:40:26.738622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:31.188904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:31.188904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:35.940813+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":186,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.728160+0000","last_change":"2024-03-26T21:39:37.779162+0000","last_active":"2024-03-26T21:40:26.728160+0000","last_peered":"2024-03-26T21:40:26.728160+0000","last_clean":"2024-03-26T21:40:26.728160+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:40:26.728160+0000","last_undegraded":"2024-03-26T21:40:26.728160+0000","last_fullsized":"2024-03-26T21:40:26.728160+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:39:37.778878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:37.778878+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:41.899300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.742738+0000","last_change":"2024-03-26T21:39:47.634422+0000","last_active":"2024-03-26T21:40:26.742738+0000","last_peered":"2024-03-26T21:40:26.742738+0000","last_clean":"2024-03-26T21:39:37.767452+0000","last_became_active":"2024-03-26T21:39:47.634422+0000","last_became_peered":"2024-03-26T21:39:47.634422+0000","last_unstale":"2024-03-26T21:40:26.742738+0000","last_undegraded":"2024-03-26T21:40:26.742738+0000","last_fullsized":"2024-03-26T21:39:47.608302+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:40.936560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:29.955684+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.936560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":542,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.736226+0000","last_change":"2024-03-26T21:40:14.873262+0000","last_active":"2024-03-26T21:40:26.736226+0000","last_peered":"2024-03-26T21:40:26.736226+0000","last_clean":"2024-03-26T21:40:26.736226+0000","last_became_active":"2024-03-26T21:36:51.716671+0000","last_became_peered":"2024-03-26T21:36:51.716671+0000","last_unstale":"2024-03-26T21:40:26.736226+0000","last_undegraded":"2024-03-26T21:40:26.736226+0000","last_fullsized":"2024-03-26T21:40:26.736226+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:14.873121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:14.873121+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:30.998405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"260'6758","reported_seq":6810,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:32.484922+0000","last_change":"2024-03-26T21:39:47.631418+0000","last_active":"2024-03-26T21:40:32.484922+0000","last_peered":"2024-03-26T21:40:32.484922+0000","last_clean":"2024-03-26T21:39:46.584716+0000","last_became_active":"2024-03-26T21:39:47.631418+0000","last_became_peered":"2024-03-26T21:39:47.631418+0000","last_unstale":"2024-03-26T21:40:32.484922+0000","last_undegraded":"2024-03-26T21:39:47.615663+0000","last_fullsized":"2024-03-26T21:39:47.615542+0000","mapping_epoch":249,"log_start":"245'2000","ondisk_log_start":"245'2000","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442892288,"num_objects":6758,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6758,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":6758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":422,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.744991+0000","last_change":"2024-03-26T21:37:52.675768+0000","last_active":"2024-03-26T21:40:26.744991+0000","last_peered":"2024-03-26T21:40:26.744991+0000","last_clean":"2024-03-26T21:40:26.744991+0000","last_became_active":"2024-03-26T21:28:41.113522+0000","last_became_peered":"2024-03-26T21:28:41.113522+0000","last_unstale":"2024-03-26T21:40:26.744991+0000","last_undegraded":"2024-03-26T21:40:26.744991+0000","last_fullsized":"2024-03-26T21:40:26.744991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:52.675543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:37:52.675543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.739971+0000","last_change":"2024-03-26T21:39:29.056180+0000","last_active":"2024-03-26T21:40:26.739971+0000","last_peered":"2024-03-26T21:40:26.739971+0000","last_clean":"2024-03-26T21:40:26.739971+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:40:26.739971+0000","last_undegraded":"2024-03-26T21:40:26.739971+0000","last_fullsized":"2024-03-26T21:40:26.739971+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:29.056054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.056054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:38.618796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":133,"reported_epoch":260,"state":"active+undersized","last_fresh":"2024-03-26T21:40:26.725657+0000","last_change":"2024-03-26T21:39:47.613548+0000","last_active":"2024-03-26T21:40:26.725657+0000","last_peered":"2024-03-26T21:40:26.725657+0000","last_clean":"2024-03-26T21:39:37.711304+0000","last_became_active":"2024-03-26T21:39:47.613548+0000","last_became_peered":"2024-03-26T21:39:47.613548+0000","last_unstale":"2024-03-26T21:40:26.725657+0000","last_undegraded":"2024-03-26T21:40:26.725657+0000","last_fullsized":"2024-03-26T21:39:47.605040+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.897125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.897125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":162,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.727954+0000","last_change":"2024-03-26T21:40:16.848860+0000","last_active":"2024-03-26T21:40:26.727954+0000","last_peered":"2024-03-26T21:40:26.727954+0000","last_clean":"2024-03-26T21:40:26.727954+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:40:26.727954+0000","last_undegraded":"2024-03-26T21:40:26.727954+0000","last_fullsized":"2024-03-26T21:40:26.727954+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:40:16.848737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.848737+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:35.069218+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"260'6779","reported_seq":6829,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:32.281856+0000","last_change":"2024-03-26T21:39:12.242170+0000","last_active":"2024-03-26T21:40:32.281856+0000","last_peered":"2024-03-26T21:40:32.281856+0000","last_clean":"2024-03-26T21:40:32.281856+0000","last_became_active":"2024-03-26T21:39:12.241837+0000","last_became_peered":"2024-03-26T21:39:12.241837+0000","last_unstale":"2024-03-26T21:40:32.281856+0000","last_undegraded":"2024-03-26T21:40:32.281856+0000","last_fullsized":"2024-03-26T21:40:32.281856+0000","mapping_epoch":240,"log_start":"246'2000","ondisk_log_start":"246'2000","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444268544,"num_objects":6779,"num_object_clones":0,"num_object_copies":20337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":459,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.750812+0000","last_change":"2024-03-26T21:40:25.429819+0000","last_active":"2024-03-26T21:40:26.750812+0000","last_peered":"2024-03-26T21:40:26.750812+0000","last_clean":"2024-03-26T21:40:26.750812+0000","last_became_active":"2024-03-26T21:36:51.698823+0000","last_became_peered":"2024-03-26T21:36:51.698823+0000","last_unstale":"2024-03-26T21:40:26.750812+0000","last_undegraded":"2024-03-26T21:40:26.750812+0000","last_fullsized":"2024-03-26T21:40:26.750812+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:25.429677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.429677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:42.947718+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":471,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.732095+0000","last_change":"2024-03-26T21:40:22.891092+0000","last_active":"2024-03-26T21:40:26.732095+0000","last_peered":"2024-03-26T21:40:26.732095+0000","last_clean":"2024-03-26T21:40:26.732095+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:40:26.732095+0000","last_undegraded":"2024-03-26T21:40:26.732095+0000","last_fullsized":"2024-03-26T21:40:26.732095+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.890287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.890287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:37.174434+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":471,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.732492+0000","last_change":"2024-03-26T21:40:20.981909+0000","last_active":"2024-03-26T21:40:26.732492+0000","last_peered":"2024-03-26T21:40:26.732492+0000","last_clean":"2024-03-26T21:40:26.732492+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:40:26.732492+0000","last_undegraded":"2024-03-26T21:40:26.732492+0000","last_fullsized":"2024-03-26T21:40:26.732492+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.981778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.981778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:24.801638+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"234'31","reported_seq":177,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726702+0000","last_change":"2024-03-26T21:39:43.869443+0000","last_active":"2024-03-26T21:40:26.726702+0000","last_peered":"2024-03-26T21:40:26.726702+0000","last_clean":"2024-03-26T21:40:26.726702+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:40:26.726702+0000","last_undegraded":"2024-03-26T21:40:26.726702+0000","last_fullsized":"2024-03-26T21:40:26.726702+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'31","last_scrub_stamp":"2024-03-26T21:39:43.869278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:43.869278+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:08.913036+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"260'6800","reported_seq":6853,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:32.512791+0000","last_change":"2024-03-26T21:39:47.619535+0000","last_active":"2024-03-26T21:40:32.512791+0000","last_peered":"2024-03-26T21:40:32.512791+0000","last_clean":"2024-03-26T21:39:46.594062+0000","last_became_active":"2024-03-26T21:39:47.619535+0000","last_became_peered":"2024-03-26T21:39:47.619535+0000","last_unstale":"2024-03-26T21:40:32.512791+0000","last_undegraded":"2024-03-26T21:39:47.608851+0000","last_fullsized":"2024-03-26T21:39:47.608752+0000","mapping_epoch":249,"log_start":"247'2100","ondisk_log_start":"247'2100","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:11.097026+0000","last_clean_scrub_stamp":"2024-03-26T21:39:11.097026+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445644800,"num_objects":6800,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6800,"num_write_kb":435200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":6800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":138,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.727852+0000","last_change":"2024-03-26T21:40:18.803154+0000","last_active":"2024-03-26T21:40:26.727852+0000","last_peered":"2024-03-26T21:40:26.727852+0000","last_clean":"2024-03-26T21:40:26.727852+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:40:26.727852+0000","last_undegraded":"2024-03-26T21:40:26.727852+0000","last_fullsized":"2024-03-26T21:40:26.727852+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:18.803008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:18.803008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:20.233354+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"250'10","reported_seq":599,"reported_epoch":260,"state":"active+undersized+degraded","last_fresh":"2024-03-26T21:40:26.752286+0000","last_change":"2024-03-26T21:39:47.618044+0000","last_active":"2024-03-26T21:40:26.752286+0000","last_peered":"2024-03-26T21:40:26.752286+0000","last_clean":"2024-03-26T21:39:45.023705+0000","last_became_active":"2024-03-26T21:39:47.618044+0000","last_became_peered":"2024-03-26T21:39:47.618044+0000","last_unstale":"2024-03-26T21:40:26.752286+0000","last_undegraded":"2024-03-26T21:39:47.607557+0000","last_fullsized":"2024-03-26T21:39:47.607431+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'9","last_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:38:57.799649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:40:31.282481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":439,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.747000+0000","last_change":"2024-03-26T21:37:35.574887+0000","last_active":"2024-03-26T21:40:26.747000+0000","last_peered":"2024-03-26T21:40:26.747000+0000","last_clean":"2024-03-26T21:40:26.747000+0000","last_became_active":"2024-03-26T21:36:51.691808+0000","last_became_peered":"2024-03-26T21:36:51.691808+0000","last_unstale":"2024-03-26T21:40:26.747000+0000","last_undegraded":"2024-03-26T21:40:26.747000+0000","last_fullsized":"2024-03-26T21:40:26.747000+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:37:35.574719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:37:35.574719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":165,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-26T21:40:26.726129+0000","last_change":"2024-03-26T21:40:25.861928+0000","last_active":"2024-03-26T21:40:26.726129+0000","last_peered":"2024-03-26T21:40:26.726129+0000","last_clean":"2024-03-26T21:40:26.726129+0000","last_became_active":"2024-03-26T21:36:51.689830+0000","last_became_peered":"2024-03-26T21:36:51.689830+0000","last_unstale":"2024-03-26T21:40:26.726129+0000","last_undegraded":"2024-03-26T21:40:26.726129+0000","last_fullsized":"2024-03-26T21:40:26.726129+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:40:25.861782+0000","last_deep_scrub":"87'15","last_deep_scrub_stamp":"2024-03-26T21:29:37.647105+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.861782+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:41:37.089062+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6956646400,"num_objects":106150,"num_object_clones":0,"num_object_copies":318450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106150,"num_write_kb":6793600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18879479808,"data_stored":18879479808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82050,"ondisk_log_size":82050,"up":39,"acting":39,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":695,"num_read_kb":690,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":83,"acting":83,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070603,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106636,"kb_used_data":1800580,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91638516,"statfs":{"total":95995035648,"available":93837840384,"internally_reserved":0,"allocated":1843793920,"data_stored":1843375240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":313391562},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364954,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1609212,"kb_used_data":1276788,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92135940,"statfs":{"total":95995035648,"available":94347202560,"internally_reserved":0,"allocated":1307430912,"data_stored":1306978686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4236524,"kb_used_data":3858276,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":89508628,"statfs":{"total":95995035648,"available":91656835072,"internally_reserved":0,"allocated":3950874624,"data_stored":3950374565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2053728,"kb_used_data":1639380,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91691424,"statfs":{"total":95995035648,"available":93892018176,"internally_reserved":0,"allocated":1678725120,"data_stored":1678241844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411502,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1639988,"kb_used_data":1293676,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92105164,"statfs":{"total":95995035648,"available":94315687936,"internally_reserved":0,"allocated":1324724224,"data_stored":1324279761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3960416,"kb_used_data":3401688,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":89784736,"statfs":{"total":95995035648,"available":91939569664,"internally_reserved":0,"allocated":3483328512,"data_stored":3482839782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":572127687},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650460,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530140,"kb_used_data":1209236,"kb_used_omap":4,"kb_used_meta":320891,"kb_avail":92215012,"statfs":{"total":95995035648,"available":94428172288,"internally_reserved":0,"allocated":1238257664,"data_stored":1237774078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":328593059},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659293,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4180620,"kb_used_data":3824580,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":89564532,"statfs":{"total":95995035648,"available":91714080768,"internally_reserved":0,"allocated":3916369920,"data_stored":3915901450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1686372352,"data_stored":1686372352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3964076032,"data_stored":3964076032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1331298304,"data_stored":1331298304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3519283200,"data_stored":3519283200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1247215616,"data_stored":1247215616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3941990400,"data_stored":3941990400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1314717696,"data_stored":1314717696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1874526208,"data_stored":1874526208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:40:34.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-26T21:40:34.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 73 16 110407 110391 94.4946 67 0.0108845 0.0105771 2024-03-26T21:40:35.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.098 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:35.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:35.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:35.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[29406]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:40:35.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[29406]: pgmap v521: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 60300/322524 objects degraded (18.696%) 2024-03-26T21:40:35.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[29406]: 9.b scrub starts 2024-03-26T21:40:35.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[40587]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:40:35.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[40587]: pgmap v521: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 60300/322524 objects degraded (18.696%) 2024-03-26T21:40:35.226 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:35 smithi055 ceph-mon[40587]: 9.b scrub starts 2024-03-26T21:40:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.254 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:35.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:35.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:35.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:35 smithi067 ceph-mon[33346]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:40:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:35 smithi067 ceph-mon[33346]: pgmap v521: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 60300/322524 objects degraded (18.696%) 2024-03-26T21:40:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:35 smithi067 ceph-mon[33346]: 9.b scrub starts 2024-03-26T21:40:35.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:35.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:35.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:35.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:35.833 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:35.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:35.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:35.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:35.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 74 16 111459 111443 94.1061 65.75 0.0108441 0.010619 2024-03-26T21:40:36.016 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:40:36.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:36.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:36.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:36.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:36 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:40:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:36 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:40:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:36 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:40:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:36 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:40:36.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:36 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:40:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:36 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:40:36.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:36.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.664 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:36.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:36.792 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:36.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:36.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:36.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 75 16 112678 112662 93.8671 76.1875 0.017057 0.0106482 2024-03-26T21:40:37.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:37.076 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:37.125 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:37.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:37.189 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:37.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: pgmap v522: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 60664/323616 objects degraded (18.746%) 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 9.0 scrub ok 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 3.1c deep-scrub starts 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 3.1c deep-scrub ok 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 9.b scrub ok 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 60664/323616 objects degraded (18.746%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[40587]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T21:40:37.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: pgmap v522: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 60664/323616 objects degraded (18.746%) 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 9.0 scrub ok 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 3.1c deep-scrub starts 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 3.1c deep-scrub ok 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 9.b scrub ok 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 60664/323616 objects degraded (18.746%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:37.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:37 smithi055 ceph-mon[29406]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T21:40:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:37.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:37.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:37.445 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:37.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: pgmap v522: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 60664/323616 objects degraded (18.746%) 2024-03-26T21:40:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 9.0 scrub ok 2024-03-26T21:40:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 3.1c deep-scrub starts 2024-03-26T21:40:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 3.1c deep-scrub ok 2024-03-26T21:40:37.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 9.b scrub ok 2024-03-26T21:40:37.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:40:37.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:40:37.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 60664/323616 objects degraded (18.746%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:37.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:37 smithi067 ceph-mon[33346]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T21:40:37.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:37.571 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:37.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:37.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:37.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:37.845 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:37.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:37.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:37.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:37.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 76 16 113733 113717 93.4996 65.9375 0.0100865 0.0106906 2024-03-26T21:40:38.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:38.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:38.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:40:38.290 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:40:38.291 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:38.291 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:40:38.291 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:40:38.291 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:38.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:38.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:38 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:40:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:38 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:40:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:38.504 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:38.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:38.629 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:38.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:38.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:38.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:38.911 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:38.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:38.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 77 16 114999 114983 93.3128 79.125 0.0140747 0.0107124 2024-03-26T21:40:39.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:39.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:39.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:39.183 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:39.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[40587]: pgmap v524: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 61667/330786 objects degraded (18.643%) 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[40587]: 4.1d scrub starts 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[40587]: 4.1d scrub ok 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[29406]: pgmap v524: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 61667/330786 objects degraded (18.643%) 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:40:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:40:39.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[29406]: 4.1d scrub starts 2024-03-26T21:40:39.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:39 smithi055 ceph-mon[29406]: 4.1d scrub ok 2024-03-26T21:40:39.317 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:39.325 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:39.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:39.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:39 smithi067 ceph-mon[33346]: pgmap v524: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 61667/330786 objects degraded (18.643%) 2024-03-26T21:40:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:39 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:40:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:39 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:40:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:39 smithi067 ceph-mon[33346]: 4.1d scrub starts 2024-03-26T21:40:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:39 smithi067 ceph-mon[33346]: 4.1d scrub ok 2024-03-26T21:40:39.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:39.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:39.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:39.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:39.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:39.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:39.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:39.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:39.998 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 78 15 116265 116250 93.1316 79.1875 0.00517207 0.0107334 2024-03-26T21:40:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.034 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:40.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:40.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:40.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:40.272 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:40:40.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[29406]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:40:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:40 smithi055 ceph-mon[40587]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T21:40:40.319 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:40:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.330 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:40.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:40:40.406 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:40:40.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:40.447 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:40:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.462 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:40.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:40 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:40:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:40 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:40:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:40 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:40:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:40 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:40:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:40 smithi067 ceph-mon[33346]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T21:40:40.490 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:40:40.523 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:40:40.525 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:40:40.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:40.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:40.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:40.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:40.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:40.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:40.999 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 79 16 117473 117457 92.9076 75.4375 0.0211978 0.0107583 2024-03-26T21:40:41.016 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:41.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:41.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:41.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:41.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: pgmap v526: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 69 MiB/s wr, 1.10k op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/667827983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:40:41.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: pgmap v526: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 69 MiB/s wr, 1.10k op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/667827983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:40:41.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:41 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:40:41.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:41.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: pgmap v526: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 796 B/s rd, 69 MiB/s wr, 1.10k op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/667827983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:40:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:41 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:40:41.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:41.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:41.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.699 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:41.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:41.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:41.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:41.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:42.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:40:42.008244+0000 min lat: 0.00115728 max lat: 1.04681 avg lat: 0.0107799 2024-03-26T21:40:42.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:40:42.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 80 16 118730 118714 92.7271 78.5625 0.00950035 0.0107799 2024-03-26T21:40:42.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:42.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:42.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:42.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:42.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:40:42.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: 3.2 deep-scrub starts 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: 3.2 deep-scrub ok 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 63768/340224 objects degraded (18.743%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[29406]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: 3.2 deep-scrub starts 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: 3.2 deep-scrub ok 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 63768/340224 objects degraded (18.743%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:42.470 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:42 smithi055 ceph-mon[40587]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: 3.2 deep-scrub starts 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: 3.2 deep-scrub ok 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 63768/340224 objects degraded (18.743%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:42 smithi067 ceph-mon[33346]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T21:40:42.502 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:42.563 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:42.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:42.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:42.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:42.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:42.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:43.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 81 16 120003 119987 92.5645 79.5625 0.00859178 0.0107988 2024-03-26T21:40:43.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:43.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:43.234 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:43.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:43.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:43.476 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:43 smithi055 ceph-mon[40587]: pgmap v527: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 737 op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:43.477 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:43 smithi055 ceph-mon[29406]: pgmap v527: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 737 op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:43 smithi067 ceph-mon[33346]: pgmap v527: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 737 op/s; 63768/340224 objects degraded (18.743%) 2024-03-26T21:40:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:43.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:43.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.778 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:43.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:40:43.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:40:43.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:43.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:44.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 82 16 121227 121211 92.3685 76.5 0.0154619 0.0108218 2024-03-26T21:40:44.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:44.035 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:44.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:44.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:44.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:44.303 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:44.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:40:44.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:40:44.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:40:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:40:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:40:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:40:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:40:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:44 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:40:44.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:44.445 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:44.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:44.589 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:44.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:44.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:44.803 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:44 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:40:44.803 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:44 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:40:44.803 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:44 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:40:44.803 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:44 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:40:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:44.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:44.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:44.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:45.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 83 16 122422 122406 92.1555 74.6875 0.0157203 0.0108468 2024-03-26T21:40:45.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:45.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:45.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.214 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:45.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:45.281 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 1 pgp_num to 1 2024-03-26T21:40:45.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:45.377 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:45.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:45.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[29406]: pgmap v529: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 66824/357558 objects degraded (18.689%) 2024-03-26T21:40:45.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:40:45.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:40:45.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2734730725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[40587]: pgmap v529: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 66824/357558 objects degraded (18.689%) 2024-03-26T21:40:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:40:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:40:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2734730725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:45.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:45 smithi067 ceph-mon[33346]: pgmap v529: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 91 MiB/s wr, 1.46k op/s; 66824/357558 objects degraded (18.689%) 2024-03-26T21:40:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:45 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:40:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:45 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:40:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:45 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2734730725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:40:45.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:45.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:40:45.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:45.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:45.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:46.000 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 84 16 123578 123562 91.9184 72.25 0.00951287 0.0108746 2024-03-26T21:40:46.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:46.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:46.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:46.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:46.317 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:46.419 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:46.461 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:46.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[40587]: 9.6 scrub starts 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[40587]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[29406]: 9.6 scrub starts 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:40:46.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[29406]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:46 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:40:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:46.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:46.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:46 smithi067 ceph-mon[33346]: 9.6 scrub starts 2024-03-26T21:40:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:46 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:40:46.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:46 smithi067 ceph-mon[33346]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T21:40:46.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:46 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:40:46.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:46 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:40:46.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:46.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:46.971 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:47.001 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 85 16 124704 124688 91.6642 70.375 0.029736 0.0109047 2024-03-26T21:40:47.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:47.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:47.165 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:47.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:47.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:47.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-mon[33346]: pgmap v531: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 806 B/s rd, 61 MiB/s wr, 970 op/s; 67201/358689 objects degraded (18.735%) 2024-03-26T21:40:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 67201/358689 objects degraded (18.735%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-mon[33346]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T21:40:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:47.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.555 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:47.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[29406]: pgmap v531: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 806 B/s rd, 61 MiB/s wr, 970 op/s; 67201/358689 objects degraded (18.735%) 2024-03-26T21:40:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 67201/358689 objects degraded (18.735%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[29406]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T21:40:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[40587]: pgmap v531: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 806 B/s rd, 61 MiB/s wr, 970 op/s; 67201/358689 objects degraded (18.735%) 2024-03-26T21:40:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 67201/358689 objects degraded (18.735%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T21:40:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[40587]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T21:40:47.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:47.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:40:47.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:47.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:47.746 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:40:47.747 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:40:47.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:47.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:47.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:47.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:48.001 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 86 16 125881 125865 91.4537 73.5625 0.014435 0.0109296 2024-03-26T21:40:48.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:48.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:48.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:48.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:48.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:48.412 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:48.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:48.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:40:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:40:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 9.6 scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 3.1 deep-scrub starts 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 3.1 deep-scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 9.6 scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 3.1 deep-scrub starts 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 3.1 deep-scrub ok 2024-03-26T21:40:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:40:48.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:40:48.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:48.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:40:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.703 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:48.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 9.6 scrub ok 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 3.1 deep-scrub starts 2024-03-26T21:40:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 3.1 deep-scrub ok 2024-03-26T21:40:48.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:40:48.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:48 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:40:48.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:48.830 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:48.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:48.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:49.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 87 16 127011 126995 91.2104 70.625 0.0251936 0.0109582 2024-03-26T21:40:49.047 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:49.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:49.085 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:49.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:49.198 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.241 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:49.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:49.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:49.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:49.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:49.494 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[40587]: pgmap v533: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 68288/366579 objects degraded (18.628%) 2024-03-26T21:40:49.494 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:40:49.494 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:40:49.494 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[29406]: pgmap v533: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 68288/366579 objects degraded (18.628%) 2024-03-26T21:40:49.494 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:40:49.495 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:49 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:40:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:49.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:49.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:49.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:49 smithi067 ceph-mon[33346]: pgmap v533: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 68288/366579 objects degraded (18.628%) 2024-03-26T21:40:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:49 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:40:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:49 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:40:49.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:49.802 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:49.880 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:49.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:49.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:50.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 88 16 128130 128114 90.9686 69.9375 0.0173465 0.0109881 2024-03-26T21:40:50.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:50.072 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:50.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:50.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:50.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:50.332 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:40:50.379 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:50.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:50.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:40:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:40:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2340303568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[40587]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2340303568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:50 smithi055 ceph-mon[29406]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:50.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:50.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:50 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:40:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:50 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:40:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:50 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2340303568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:50 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:40:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:50 smithi067 ceph-mon[33346]: pgmap v534: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:50.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:50.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:50.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:50.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:51.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 89 16 129435 129419 90.8628 81.5625 0.00577449 0.0110011 2024-03-26T21:40:51.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:51.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:51.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:51.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:51.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: 6.11 deep-scrub starts 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: 6.11 deep-scrub ok 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[29406]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: 6.11 deep-scrub starts 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: 6.11 deep-scrub ok 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:40:51.420 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:51.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:40:51.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:51 smithi055 ceph-mon[40587]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T21:40:51.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:51.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:51.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:51.715 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: 6.11 deep-scrub starts 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: 6.11 deep-scrub ok 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:40:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:51 smithi067 ceph-mon[33346]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T21:40:51.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:51.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:51.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:51.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:52.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:52.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total time run: 90.0063 2024-03-26T21:40:52.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total writes made: 130648 2024-03-26T21:40:52.005 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Write size: 65536 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object size: 65536 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Bandwidth (MB/sec): 90.7215 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Bandwidth: 32.4078 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max bandwidth (MB/sec): 220 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min bandwidth (MB/sec): 31.375 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average IOPS: 1451 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev IOPS: 518.524 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max IOPS: 3520 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min IOPS: 502 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average Latency(s): 0.0110189 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Latency(s): 0.0145101 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max latency(s): 1.04681 2024-03-26T21:40:52.006 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min latency(s): 0.00115728 2024-03-26T21:40:52.010 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T21:40:52.044 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:52.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.184 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:52.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:52.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:52.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 5, 6, 4] 2024-03-26T21:40:52.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:40:52.418 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-26T21:40:52.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd out 3 2024-03-26T21:40:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:52.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:52.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:52.640 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[29406]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 815 B/s rd, 59 MiB/s wr, 948 op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:52.640 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 70505/376536 objects degraded (18.725%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:52.640 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[29406]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T21:40:52.640 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[40587]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 815 B/s rd, 59 MiB/s wr, 948 op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:52.641 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 70505/376536 objects degraded (18.725%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:52.641 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:52 smithi055 ceph-mon[40587]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T21:40:52.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:52 smithi067 ceph-mon[33346]: pgmap v536: 148 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 32 active+undersized, 85 active+clean; 7.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 815 B/s rd, 59 MiB/s wr, 948 op/s; 70505/376536 objects degraded (18.725%) 2024-03-26T21:40:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 70505/376536 objects degraded (18.725%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:52 smithi067 ceph-mon[33346]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T21:40:52.758 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:52.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:52.878 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:40:52.919 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:40:52.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:52.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:52.962 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:40:53.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:40:53.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:53.042 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:40:53.077 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:40:53.118 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:53.120 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:40:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:40:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:40:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:40:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:40:53.123 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:40:53.124 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:40:53.124 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:40:53.124 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:40:53.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:40:53.221 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:53.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:53.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:53.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:53.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:53.552 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/196673617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:53.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/196673617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:53.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:53.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/196673617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:40:53.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:53.830 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:53.932 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:53.974 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:53.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:54.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:54.120 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:54.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:54.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:54.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:54.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:54.534 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:54.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:54.677 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:54.683 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:54 smithi055 ceph-mon[29406]: pgmap v538: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.49k op/s; 72728/389409 objects degraded (18.677%) 2024-03-26T21:40:54.683 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:54 smithi055 ceph-mon[40587]: pgmap v538: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.49k op/s; 72728/389409 objects degraded (18.677%) 2024-03-26T21:40:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:54 smithi067 ceph-mon[33346]: pgmap v538: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 87 MiB/s wr, 1.49k op/s; 72728/389409 objects degraded (18.677%) 2024-03-26T21:40:54.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:54.805 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:54.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:54.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:40:55.023 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:40:55.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:55.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:55.203 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:40:55.203 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:40:55.203 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:40:55.204 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:40:55.204 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:40:55.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:55.212 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:55.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:55.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:55.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:55.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:55.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:55.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.768 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:55.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:55.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:55.921 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:55.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:56.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:56.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:56.101 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:40:56.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:40:56.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:56.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:56.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:56.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:56.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:56.603 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:56.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:56.725 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:56 smithi055 ceph-mon[29406]: pgmap v539: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.17k op/s; 72870/389835 objects degraded (18.693%) 2024-03-26T21:40:56.725 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:56 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:56.725 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:56 smithi055 ceph-mon[40587]: pgmap v539: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.17k op/s; 72870/389835 objects degraded (18.693%) 2024-03-26T21:40:56.726 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:56 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:56.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:56.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:56.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:56.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:56 smithi067 ceph-mon[33346]: pgmap v539: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.17k op/s; 72870/389835 objects degraded (18.693%) 2024-03-26T21:40:56.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:56 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:40:57.024 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:40:57.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:57.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:57.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:57.336 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:57.376 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:57.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:57.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:40:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:40:57.508 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:57.609 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:57.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:57.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:57.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:40:57.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:57.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:40:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 72870/389835 objects degraded (18.693%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:57.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:58.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:58.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 72870/389835 objects degraded (18.693%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:58.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 72870/389835 objects degraded (18.693%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:40:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:40:58.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:58.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:58.318 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:58.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:58.466 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:58.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:40:58.597 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:40:58.699 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:40:58.739 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:40:58.841 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:58.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:58.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:58.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[40587]: pgmap v540: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.10k op/s; 71793/382197 objects degraded (18.784%) 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[29406]: pgmap v540: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.10k op/s; 71793/382197 objects degraded (18.784%) 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:58.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:58 smithi067 ceph-mon[33346]: pgmap v540: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.10k op/s; 71793/382197 objects degraded (18.784%) 2024-03-26T21:40:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:40:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:40:58.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:40:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:40:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:40:59.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:59.143 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:59.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:40:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:59.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:40:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:40:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:40:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:40:59.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:59.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:59.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:40:59.726 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:40:59.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:40:59.863 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:40:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:40:59.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:40:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:40:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:40:59.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:00.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:00.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:00.276 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:00.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:00.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:00.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:00.633 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:00.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:00.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:00.799 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:00.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:00.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:00.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:00.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:00 smithi067 ceph-mon[33346]: pgmap v541: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 1.34k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:01.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:01.064 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:00 smithi055 ceph-mon[29406]: pgmap v541: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 1.34k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:00 smithi055 ceph-mon[40587]: pgmap v541: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 1.34k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:01.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:01.190 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:01.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:01.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:01.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:01.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:01.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:01.611 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:01.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:01.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:01.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:01.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:01.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:01.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:02.010 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:02.097 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:02.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:02.156 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:02.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:02.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:02.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:02.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:02.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:02.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:02.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:02.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:02.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:02.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:02.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:02.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:02 smithi067 ceph-mon[33346]: pgmap v542: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.24k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 69461/371430 objects degraded (18.701%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[29406]: pgmap v542: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.24k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 69461/371430 objects degraded (18.701%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[40587]: pgmap v542: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.24k op/s; 69461/371430 objects degraded (18.701%) 2024-03-26T21:41:03.062 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 69461/371430 objects degraded (18.701%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:03.063 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:03.063 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:03.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:03.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:03.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:03.246 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:03.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:03.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:03.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:03.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:03.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:03.658 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:03.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:03.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:03.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:04.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:04.052 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:04.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:04.195 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:04.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:04.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:04.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.487 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:04.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:04.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:04.634 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:04.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:04.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:04 smithi055 ceph-mon[40587]: pgmap v543: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.41k op/s; 66426/354153 objects degraded (18.756%) 2024-03-26T21:41:04.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:04 smithi055 ceph-mon[29406]: pgmap v543: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.41k op/s; 66426/354153 objects degraded (18.756%) 2024-03-26T21:41:04.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:04.893 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:04.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:04 smithi067 ceph-mon[33346]: pgmap v543: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.41k op/s; 66426/354153 objects degraded (18.756%) 2024-03-26T21:41:04.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:05.025 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:05.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:05.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:05.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:41:05.082+0000 7fb1ff446700 -1 mon.a@0(leader).osd e268 definitely_dead 0 2024-03-26T21:41:05.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:05.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:05.405 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:05.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:05.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:05.670 INFO:teuthology.orchestra.run.smithi055.stderr:marked out osd.3. 2024-03-26T21:41:05.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:05.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1107153856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[40587]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1107153856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:05 smithi055 ceph-mon[29406]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T21:41:05.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:05.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:05.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:05.967 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:05.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:05 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1107153856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:05 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T21:41:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:05 smithi067 ceph-mon[33346]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T21:41:06.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:06.093 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:06.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:06.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:06.344 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:06.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:06.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:06.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:06.575 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.597 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:06.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:06.671 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:41:06.698 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:06.748 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:06.850 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:06.874 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:06.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:06.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: pgmap v544: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 1.9 MiB/s wr, 1.07k op/s; 66049/353022 objects degraded (18.710%) 2024-03-26T21:41:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T21:41:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: osdmap e269: 8 total, 7 up, 7 in 2024-03-26T21:41:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2322744159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:06 smithi067 ceph-mon[33346]: osdmap e270: 8 total, 7 up, 7 in 2024-03-26T21:41:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: pgmap v544: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 1.9 MiB/s wr, 1.07k op/s; 66049/353022 objects degraded (18.710%) 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: osdmap e269: 8 total, 7 up, 7 in 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2322744159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[29406]: osdmap e270: 8 total, 7 up, 7 in 2024-03-26T21:41:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: pgmap v544: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 1.9 MiB/s wr, 1.07k op/s; 66049/353022 objects degraded (18.710%) 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: osdmap e269: 8 total, 7 up, 7 in 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2322744159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:07.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:06 smithi055 ceph-mon[40587]: osdmap e270: 8 total, 7 up, 7 in 2024-03-26T21:41:07.102 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:07.126 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:07.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:07.252 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:07.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:07.377 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:07.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:07.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:07.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:07.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 66049/353022 objects degraded (18.710%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:07.723 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[29406]: osdmap e271: 8 total, 7 up, 7 in 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 66049/353022 objects degraded (18.710%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:07.724 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:07 smithi055 ceph-mon[40587]: osdmap e271: 8 total, 7 up, 7 in 2024-03-26T21:41:07.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:07.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:07.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:07.918 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:07.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 66049/353022 objects degraded (18.710%), 30 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:07 smithi067 ceph-mon[33346]: osdmap e271: 8 total, 7 up, 7 in 2024-03-26T21:41:08.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:08.058 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:08.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:08.195 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:08.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.330 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:08.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:08.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:08.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:08.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:08.602 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:08.704 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:08.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:08.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:08.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:08.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:08.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: pgmap v547: 148 pgs: 12 remapped+peering, 4 unknown, 13 peering, 22 active+undersized+degraded, 26 active+undersized, 71 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.12k op/s; 43328/344478 objects degraded (12.578%); 60 B/s, 2 objects/s recovering 2024-03-26T21:41:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 268'9378 2024-03-26T21:41:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 268'9204 2024-03-26T21:41:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 268'9101 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 268'9320 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 268'9038 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 268'9098 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:08 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: pgmap v547: 148 pgs: 12 remapped+peering, 4 unknown, 13 peering, 22 active+undersized+degraded, 26 active+undersized, 71 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.12k op/s; 43328/344478 objects degraded (12.578%); 60 B/s, 2 objects/s recovering 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 268'9378 2024-03-26T21:41:08.990 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 268'9204 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 268'9101 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 268'9320 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 268'9038 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 268'9098 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: pgmap v547: 148 pgs: 12 remapped+peering, 4 unknown, 13 peering, 22 active+undersized+degraded, 26 active+undersized, 71 active+clean; 7.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.12k op/s; 43328/344478 objects degraded (12.578%); 60 B/s, 2 objects/s recovering 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.3 starting backfill to osd.5 from (0'0,0'0] MAX to 268'9378 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 268'9204 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 268'9101 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 268'9320 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 268'9038 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 268'9098 2024-03-26T21:41:08.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T21:41:09.025 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:09.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:09.258 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.303 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:09.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:09.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:09.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:09.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:09.579 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:09.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:09.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:09.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:09.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:09.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:09.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:09 smithi067 ceph-mon[33346]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:10.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:09 smithi055 ceph-mon[40587]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:10.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:09 smithi055 ceph-mon[29406]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:10.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:10.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:10.226 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:10.264 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:10.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.419 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:10.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:10.520 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:10.561 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:10.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:10.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.837 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:10.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:10.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:10.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:10.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:10 smithi067 ceph-mon[33346]: pgmap v549: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:11.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:10 smithi055 ceph-mon[29406]: pgmap v549: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:11.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:10 smithi055 ceph-mon[40587]: pgmap v549: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:11.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:11.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:11.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.241 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:11.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:11.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:11.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:11.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:11.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:11.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:11.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:11.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:11.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:11.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:12.005 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 42336/340032 objects degraded (12.451%), 18 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:12.005 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 42336/340032 objects degraded (12.451%), 18 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:12.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:12.086 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:12.133 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:41:12.184 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:41:12.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:12.233 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:41:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 42336/340032 objects degraded (12.451%), 18 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:12.275 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:41:12.314 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:41:12.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:12.357 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:41:12.377 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:12.408 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:41:12.413 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:41:12.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:12.508 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:12.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:12.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:12.790 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:12.790 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:41:12.790 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:41:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:12.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:12.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:12.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:13.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[29406]: pgmap v550: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[29406]: osdmap e272: 8 total, 7 up, 7 in 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3326918611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[40587]: pgmap v550: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[40587]: osdmap e272: 8 total, 7 up, 7 in 2024-03-26T21:41:13.038 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3326918611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:13.074 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:13.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:13.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:12 smithi067 ceph-mon[33346]: pgmap v550: 148 pgs: 11 active+clean+remapped, 69 active+clean, 21 active+undersized, 17 active+undersized+degraded, 13 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 4 unknown, 12 remapped+peering; 6.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 42336/340032 objects degraded (12.451%); 14126/340032 objects misplaced (4.154%); 211 B/s, 9 objects/s recovering 2024-03-26T21:41:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:12 smithi067 ceph-mon[33346]: osdmap e272: 8 total, 7 up, 7 in 2024-03-26T21:41:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3326918611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:13.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:13.361 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:13.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:13.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:41:13.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:13.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:13.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:13.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:13.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:13.894 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:13.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:13.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:14.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:14.173 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:13 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 9 pgs peering) 2024-03-26T21:41:14.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:13 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 9 pgs peering) 2024-03-26T21:41:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:13 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 9 pgs peering) 2024-03-26T21:41:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.317 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:14.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:14.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:14.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:41:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:14.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:14.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:14.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:14.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:14.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [7] live_osds: [0, 1, 2, 3, 5, 6, 4] 2024-03-26T21:41:14.782 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-26T21:41:14.782 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-03-26T21:41:14.782 DEBUG:teuthology.orchestra.run.smithi067:osd.7> sudo journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T21:41:14.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl start ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7 2024-03-26T21:41:14.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:14.885 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:14.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:41:14.952 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:41:14.968 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:14.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:15.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:15.116 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:14 smithi067 ceph-mon[33346]: pgmap v552: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 593 B/s rd, 0 B/s wr, 1.08k op/s; 60893/326115 objects degraded (18.672%); 53447/326115 objects misplaced (16.389%); 12 MiB/s, 210 objects/s recovering 2024-03-26T21:41:15.116 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:14 smithi067 systemd[1]: Starting Ceph osd.7 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:41:15.124 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:14 smithi055 ceph-mon[40587]: pgmap v552: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 593 B/s rd, 0 B/s wr, 1.08k op/s; 60893/326115 objects degraded (18.672%); 53447/326115 objects misplaced (16.389%); 12 MiB/s, 210 objects/s recovering 2024-03-26T21:41:15.124 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:14 smithi055 ceph-mon[29406]: pgmap v552: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 593 B/s rd, 0 B/s wr, 1.08k op/s; 60893/326115 objects degraded (18.672%); 53447/326115 objects misplaced (16.389%); 12 MiB/s, 210 objects/s recovering 2024-03-26T21:41:15.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:15.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:15.293 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:15.336 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:15.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:15.467 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:15.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:15 smithi067 podman[116658]: 2024-03-26 21:41:15.029924621 +0000 UTC m=+0.025355175 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:41:15.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:15 smithi067 podman[116658]: 2024-03-26 21:41:15.16962154 +0000 UTC m=+0.165052119 container create 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T21:41:15.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:15.604 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:15.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:15.744 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:15.744 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:15 smithi067 podman[116658]: 2024-03-26 21:41:15.746526292 +0000 UTC m=+0.741956838 container init 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T21:41:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:15.782 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:15.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:15.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:15.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:15.985 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:16.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:16.126 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:15 smithi067 podman[116658]: 2024-03-26 21:41:15.752475773 +0000 UTC m=+0.747906305 container start 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T21:41:16.127 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:15 smithi067 podman[116658]: 2024-03-26 21:41:15.923445584 +0000 UTC m=+0.918876123 container attach 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0) 2024-03-26T21:41:16.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:16.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:16.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.463 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:16.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:16.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:16.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:16.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T21:41:16.740 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 bash[116658]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate[116672]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T21:41:16.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:16 smithi067 podman[116875]: 2024-03-26 21:41:16.567478506 +0000 UTC m=+0.036420299 container died 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T21:41:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.747 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:16.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:16.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:16.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:16.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:16.948 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[40587]: pgmap v553: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 851 op/s; 60203/324045 objects degraded (18.579%); 53108/324045 objects misplaced (16.389%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:41:16.948 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:16.948 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 60203/324045 objects degraded (18.579%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:16.949 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[29406]: pgmap v553: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 851 op/s; 60203/324045 objects degraded (18.579%); 53108/324045 objects misplaced (16.389%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:41:16.949 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:16.949 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:16 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 60203/324045 objects degraded (18.579%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:16.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:41:16.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:17.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:17.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:17.142 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-mon[33346]: pgmap v553: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 851 op/s; 60203/324045 objects degraded (18.579%); 53108/324045 objects misplaced (16.389%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:41:17.143 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:17.143 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:16 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 60203/324045 objects degraded (18.579%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:17.157 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:17.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:17.302 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:17.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:17.448 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:17.489 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 podman[116875]: 2024-03-26 21:41:17.327480074 +0000 UTC m=+0.796421847 container remove 2ddd015be7a899a037e4e6f2370319ffdbfe5aa6ad0bf087326c924aa68c8cd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:41:17.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:17.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:17.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:17.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:17.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:17.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:17.883 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 podman[116984]: 2024-03-26 21:41:17.47896253 +0000 UTC m=+0.024723574 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:41:17.883 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 podman[116984]: 2024-03-26 21:41:17.621435122 +0000 UTC m=+0.167196211 container create 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:41:17.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:17.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:41:17.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:17.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:18.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:18.104 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:18.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:18.239 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 podman[116984]: 2024-03-26 21:41:17.890812054 +0000 UTC m=+0.436573094 container init 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS) 2024-03-26T21:41:18.239 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 podman[116984]: 2024-03-26 21:41:17.896847018 +0000 UTC m=+0.442608050 container start 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T21:41:18.239 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 ceph-osd[117037]: -- 172.21.15.67:0/37524312 <== mon.1 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561af9099680 con 0x561af9063000 2024-03-26T21:41:18.239 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:17 smithi067 bash[116984]: 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 2024-03-26T21:41:18.240 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:18 smithi067 systemd[1]: Started Ceph osd.7 for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:41:18.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:18.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:41:18.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:18.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:18.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:18.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:18.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:18.700 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:18.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:18.739 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:18 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:41:18.439+0000 7fe259e2c7c0 -1 Falling back to public interface 2024-03-26T21:41:18.800 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:41:18.800 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:41:18.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:18.867 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:18.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:18.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:19.010 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:41:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:19.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:19.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:19.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:19.253 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[29406]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 931 B/s rd, 0 B/s wr, 1.04k op/s; 57967/313722 objects degraded (18.477%); 50060/313722 objects misplaced (15.957%); 19 MiB/s, 307 objects/s recovering 2024-03-26T21:41:19.253 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/224654926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.253 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.253 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:19.254 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[40587]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 931 B/s rd, 0 B/s wr, 1.04k op/s; 57967/313722 objects degraded (18.477%); 50060/313722 objects misplaced (15.957%); 19 MiB/s, 307 objects/s recovering 2024-03-26T21:41:19.254 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/224654926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.254 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:19 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:19.297 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:19.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:19.430 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:19 smithi067 ceph-mon[33346]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 931 B/s rd, 0 B/s wr, 1.04k op/s; 57967/313722 objects degraded (18.477%); 50060/313722 objects misplaced (15.957%); 19 MiB/s, 307 objects/s recovering 2024-03-26T21:41:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/224654926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:19 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:41:19.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:19.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:19.709 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:19.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:19.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:19.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:19.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:41:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:19.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:19.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:20.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:20.142 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:20.191 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:20 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:20.191 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:20 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:41:20.191 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:20 smithi067 ceph-mon[33346]: osdmap e273: 8 total, 7 up, 7 in 2024-03-26T21:41:20.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:20.275 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:41:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[29406]: osdmap e273: 8 total, 7 up, 7 in 2024-03-26T21:41:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:41:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:20 smithi055 ceph-mon[40587]: osdmap e273: 8 total, 7 up, 7 in 2024-03-26T21:41:20.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:20.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-26T21:41:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-26T21:41:20.526 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:20.577 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:20.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:20.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:20.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:20.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:20.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:21.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:21.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:21.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:21.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[29406]: pgmap v556: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 23 MiB/s, 375 objects/s recovering 2024-03-26T21:41:21.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:41:21.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:41:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[40587]: pgmap v556: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 23 MiB/s, 375 objects/s recovering 2024-03-26T21:41:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:41:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:21 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:41:21.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:21.404 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:21 smithi067 ceph-mon[33346]: pgmap v556: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 23 MiB/s, 375 objects/s recovering 2024-03-26T21:41:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:21 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:41:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:21 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:41:21.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:21.553 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:21.604 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:21.655 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:21.715 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:21.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T21:41:21.824 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-26T21:41:21.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:21.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:21.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:22.001 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:22.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:22.146 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:22.185 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 56572/307515 objects degraded (18.397%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:22.185 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:22 smithi055 ceph-mon[40587]: osdmap e274: 8 total, 7 up, 7 in 2024-03-26T21:41:22.185 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 56572/307515 objects degraded (18.397%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:22.185 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:22 smithi055 ceph-mon[29406]: osdmap e274: 8 total, 7 up, 7 in 2024-03-26T21:41:22.239 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:22 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:41:22.031+0000 7fe259e2c7c0 -1 osd.7 248 log_to_monitors true 2024-03-26T21:41:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 56572/307515 objects degraded (18.397%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:22.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:22 smithi067 ceph-mon[33346]: osdmap e274: 8 total, 7 up, 7 in 2024-03-26T21:41:22.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.300 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:22.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:22.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:22.432 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:22.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:22.550 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:41:22.575 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:22.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:22.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:22.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:22.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:23.006 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:41:23.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:23.083 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:41:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:23.111 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:41:23.140 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[29406]: pgmap v557: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[29406]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/666241150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[40587]: pgmap v557: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[40587]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.141 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/666241150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:23.141 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:41:23.166 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:41:23.193 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:41:23.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:23.214 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:41:23.214 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:41:23.214 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:41:23.214 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:41:23.214 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:41:23.215 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:41:23.215 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:41:23.215 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:41:23.215 INFO:teuthology.orchestra.run.smithi055.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-26T21:41:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:23.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:23.361 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:23.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:23 smithi067 ceph-mon[33346]: pgmap v557: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 6.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 56572/307515 objects degraded (18.397%); 49354/307515 objects misplaced (16.049%); 13 MiB/s, 205 objects/s recovering 2024-03-26T21:41:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:23 smithi067 ceph-mon[33346]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:23 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T21:41:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/666241150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:23.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:23.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:23.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:23.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:23.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:23.918 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:23.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:23.989 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:41:23 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:41:23.603+0000 7fe24727c700 -1 osd.7 275 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T21:41:24.020 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:24.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:24.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:24.198 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:24.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: osdmap e275: 8 total, 7 up, 7 in 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[29406]: from='osd.7 ' entity='osd.7' 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: osdmap e275: 8 total, 7 up, 7 in 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:41:24.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:41:24.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:24 smithi055 ceph-mon[40587]: from='osd.7 ' entity='osd.7' 2024-03-26T21:41:24.344 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:24.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: from='osd.7 v2:172.21.15.67:6812/650373445' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: osdmap e275: 8 total, 7 up, 7 in 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:41:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:24 smithi067 ceph-mon[33346]: from='osd.7 ' entity='osd.7' 2024-03-26T21:41:24.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:24.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:24.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:24.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:24.874 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:24.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T21:41:24.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T21:41:24.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:25.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:25.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:25.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:25.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:25.261 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:25.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:25.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:25.468 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:25.468 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 53581/292554 objects degraded (18.315%); 45937/292554 objects misplaced (15.702%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: osd.7 v2:172.21.15.67:6812/650373445 boot 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 53581/292554 objects degraded (18.315%); 45937/292554 objects misplaced (15.702%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: osd.7 v2:172.21.15.67:6812/650373445 boot 2024-03-26T21:41:25.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T21:41:25.470 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:25 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 53581/292554 objects degraded (18.315%); 45937/292554 objects misplaced (15.702%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: osd.7 v2:172.21.15.67:6812/650373445 boot 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T21:41:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:25 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T21:41:25.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:25.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:25.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:25.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:25.750 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:25.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:25.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:25.806 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:41:25.807 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:41:25.807 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:41:25.807 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:41:25.807 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:41:25.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:25.920 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:25.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:26.022 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:26.052 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:26.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:26.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:26.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:41:26.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 9.c continuing backfill to osd.0 from (258'5593,275'10238] MIN to 275'10238 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 9.c starting backfill to osd.7 from (0'0,248'3097] MAX to 275'10238 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10499 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 9.7 continuing backfill to osd.2 from (258'5586,275'10250] 9:e4fd327a:::benchmark_data_smithi055_135863_object60851:head to 275'10250 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[29406]: 9.7 starting backfill to osd.7 from (0'0,248'3185] MAX to 275'10250 2024-03-26T21:41:26.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 9.c continuing backfill to osd.0 from (258'5593,275'10238] MIN to 275'10238 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 9.c starting backfill to osd.7 from (0'0,248'3097] MAX to 275'10238 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10499 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 9.7 continuing backfill to osd.2 from (258'5586,275'10250] 9:e4fd327a:::benchmark_data_smithi055_135863_object60851:head to 275'10250 2024-03-26T21:41:26.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:26 smithi055 ceph-mon[40587]: 9.7 starting backfill to osd.7 from (0'0,248'3185] MAX to 275'10250 2024-03-26T21:41:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.318 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:26.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:26.419 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:26.448 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:41:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:41:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 9.c continuing backfill to osd.0 from (258'5593,275'10238] MIN to 275'10238 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 9.c starting backfill to osd.7 from (0'0,248'3097] MAX to 275'10238 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10499 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 9.7 continuing backfill to osd.2 from (258'5586,275'10250] 9:e4fd327a:::benchmark_data_smithi055_135863_object60851:head to 275'10250 2024-03-26T21:41:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:26 smithi067 ceph-mon[33346]: 9.7 starting backfill to osd.7 from (0'0,248'3185] MAX to 275'10250 2024-03-26T21:41:26.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:26.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:26.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:26.817 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:26.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:41:26.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:26.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:26.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:41:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:26.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:27.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.134 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:27.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:27.155 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: pgmap v562: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 940 op/s; 52926/290589 objects degraded (18.213%); 45598/290589 objects misplaced (15.692%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:27.155 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.1 continuing backfill to osd.4 from (257'5400,275'10481] MIN to 275'10481 2024-03-26T21:41:27.155 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.1 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10481 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.3 continuing backfill to osd.5 from (260'6531,275'10565] MIN to 275'10565 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.3 starting backfill to osd.7 from (0'0,248'3323] MAX to 275'10565 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10532 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.a starting backfill to osd.7 from (0'0,248'3259] MAX to 275'10487 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.9 starting backfill to osd.7 from (0'0,248'3275] MAX to 275'10455 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: 9.5 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10376 2024-03-26T21:41:27.156 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 52926/290589 objects degraded (18.213%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: pgmap v562: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 940 op/s; 52926/290589 objects degraded (18.213%); 45598/290589 objects misplaced (15.692%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.1 continuing backfill to osd.4 from (257'5400,275'10481] MIN to 275'10481 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.1 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10481 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.3 continuing backfill to osd.5 from (260'6531,275'10565] MIN to 275'10565 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.3 starting backfill to osd.7 from (0'0,248'3323] MAX to 275'10565 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10532 2024-03-26T21:41:27.191 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.a starting backfill to osd.7 from (0'0,248'3259] MAX to 275'10487 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.9 starting backfill to osd.7 from (0'0,248'3275] MAX to 275'10455 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: 9.5 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10376 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 52926/290589 objects degraded (18.213%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: pgmap v562: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 26 active+clean+remapped, 78 active+clean, 21 active+undersized, 17 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 940 op/s; 52926/290589 objects degraded (18.213%); 45598/290589 objects misplaced (15.692%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.1 continuing backfill to osd.4 from (257'5400,275'10481] MIN to 275'10481 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.1 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10481 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.3 continuing backfill to osd.5 from (260'6531,275'10565] MIN to 275'10565 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.3 starting backfill to osd.7 from (0'0,248'3323] MAX to 275'10565 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 275'10532 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.a starting backfill to osd.7 from (0'0,248'3259] MAX to 275'10487 2024-03-26T21:41:27.192 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T21:41:27.193 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.9 starting backfill to osd.7 from (0'0,248'3275] MAX to 275'10455 2024-03-26T21:41:27.193 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: 9.5 starting backfill to osd.7 from (0'0,248'3169] MAX to 275'10376 2024-03-26T21:41:27.193 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 52926/290589 objects degraded (18.213%), 21 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-26T21:41:27.202 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:41:27.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:27.297 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:27.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:27.438 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:27.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:27.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:27.576 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:27.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:27.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:27.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:27.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:27.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:41:27.948 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:27.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:28.019 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:28.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:28.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:28.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:28.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:28.393 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[40587]: 9.e starting backfill to osd.7 from (0'0,248'3109] MAX to 275'10248 2024-03-26T21:41:28.393 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[40587]: 9.d starting backfill to osd.7 from (0'0,248'3111] MAX to 275'10385 2024-03-26T21:41:28.393 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:28.394 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[29406]: 9.e starting backfill to osd.7 from (0'0,248'3109] MAX to 275'10248 2024-03-26T21:41:28.394 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[29406]: 9.d starting backfill to osd.7 from (0'0,248'3111] MAX to 275'10385 2024-03-26T21:41:28.394 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:28.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:28 smithi067 ceph-mon[33346]: 9.e starting backfill to osd.7 from (0'0,248'3109] MAX to 275'10248 2024-03-26T21:41:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:28 smithi067 ceph-mon[33346]: 9.d starting backfill to osd.7 from (0'0,248'3111] MAX to 275'10385 2024-03-26T21:41:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:28.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:28.579 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:28.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:28.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:28.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.867 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:28.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:28.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:28.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:29.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:29.147 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:41:29.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: pgmap v565: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 18 active+clean+remapped, 96 active+clean, 15 active+undersized, 9 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 51079/284919 objects degraded (17.928%); 41124/284919 objects misplaced (14.434%); 54 MiB/s, 864 objects/s recovering 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: 6.6 scrub starts 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: 6.6 scrub ok 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1816903070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:29 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.277 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: pgmap v565: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 18 active+clean+remapped, 96 active+clean, 15 active+undersized, 9 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 51079/284919 objects degraded (17.928%); 41124/284919 objects misplaced (14.434%); 54 MiB/s, 864 objects/s recovering 2024-03-26T21:41:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: 6.6 scrub starts 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: 6.6 scrub ok 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1816903070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: pgmap v565: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 18 active+clean+remapped, 96 active+clean, 15 active+undersized, 9 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_toofull; 5.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 51079/284919 objects degraded (17.928%); 41124/284919 objects misplaced (14.434%); 54 MiB/s, 864 objects/s recovering 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: 6.6 scrub starts 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: 6.6 scrub ok 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1816903070' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:29 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:41:29.317 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:29.319 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:41:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:41:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.25220.0:169126 9.c e279/276 9:39685d2a:::benchmark_data_smithi055_135863_object38475:head v 279'10379, mlcod=279'10377)", 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:29.321325+0000", 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0041221770000000003, 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041538970000000001, 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321325+0000", 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321325+0000", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321325+0000", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321344+0000", 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8142999999999999e-05 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321353+0000", 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4720000000000001e-06 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321361+0000", 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0509999999999997e-06 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:41:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321411+0000", 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0179999999999997e-05 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.321433+0000", 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1784999999999999e-05 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.25220.0:169118 9.3 e279/276 9:c94ce67f:::benchmark_data_smithi055_135863_object38467:head v 279'10693, mlcod=279'10691)", 2024-03-26T21:41:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:29.322331+0000", 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0031155390000000001, 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032316049999999998, 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322331+0000", 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322331+0000", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322331+0000", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4699999999999998e-07 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322358+0000", 2024-03-26T21:41:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6115e-05 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322364+0000", 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.28e-06 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322369+0000", 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1590000000000001e-06 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322474+0000", 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105235 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322497+0000", 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2755000000000002e-05 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.25220.0:169123 9.e e279/277 9:7806933a:::benchmark_data_smithi055_135863_object38472:head v 279'10387, mlcod=279'10385)", 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:29.322378+0000", 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0030687729999999999, 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032545249999999999, 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322378+0000", 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322378+0000", 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322378+0000", 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4600000000000001e-07 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:41:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322381+0000", 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8399999999999999e-06 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322385+0000", 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1760000000000003e-06 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322389+0000", 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9400000000000004e-06 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322503+0000", 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011356 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.322522+0000", 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8828e-05 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.25220.0:169124 9.3 e279/276 9:c999c457:::benchmark_data_smithi055_135863_object38473:head v 279'10694, mlcod=279'10691)", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:29.324021+0000", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0014252570000000001, 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001678351, 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324021+0000", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324021+0000", 2024-03-26T21:41:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324022+0000", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1999999999999998e-07 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324034+0000", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1337e-05 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324086+0000", 2024-03-26T21:41:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2575999999999999e-05 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.324094+0000", 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8320000000000006e-06 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325447+0000", 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00135345 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:41:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325469+0000", 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1265999999999999e-05 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.25220.0:169128 9.7 e279/276 9:eb1920d9:::benchmark_data_smithi055_135863_object38477:head v 279'10408, mlcod=279'10402)", 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:29.325004+0000", 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.00044287500000000002, 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076043900000000002, 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:41:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325004+0000", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325004+0000", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325005+0000", 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5400000000000001e-07 2024-03-26T21:41:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325011+0000", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2190000000000001e-06 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325032+0000", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1178999999999998e-05 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325039+0000", 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1799999999999999e-06 2024-03-26T21:41:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325102+0000", 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3145999999999995e-05 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:29.325122+0000", 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9559e-05 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:41:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:41:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 5 2024-03-26T21:41:29.339 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:41:29.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:29.418 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:29.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.552 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:29.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:29.653 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:29.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:29.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:29.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:29.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:29.965 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:29.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:30.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:30.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:30.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:30.338 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:30.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:30.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:30 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:41:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:30 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:41:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:30 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:41:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:30 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:30 smithi067 ceph-mon[33346]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T21:41:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:30.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:30.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:41:30.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:41:30.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[40587]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:41:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:30 smithi055 ceph-mon[29406]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T21:41:30.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:30.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:30.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:30.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:30.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:30.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:30.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:30.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:31.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:31.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:31.273 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:31.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:31.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:31.436 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 858 objects/s recovering 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T21:41:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:31 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:31.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:31.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:31.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 858 objects/s recovering 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 858 objects/s recovering 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:41:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T21:41:31.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:31.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:31 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:31.571 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:31.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.697 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:31.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:31.798 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:31.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:31.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:31.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:32.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:32.108 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:32.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.249 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:32.350 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:32.378 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:32.479 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:32 smithi067 ceph-mon[33346]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T21:41:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 50516/282168 objects degraded (17.903%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:32 smithi067 ceph-mon[33346]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T21:41:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:32.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[29406]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T21:41:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 50516/282168 objects degraded (17.903%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[29406]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T21:41:32.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[40587]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 50516/282168 objects degraded (17.903%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[40587]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:41:32.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:32.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.655 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:32.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:32.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:32.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:32.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:32.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:33.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:33.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:33.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:33.193 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:33.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:33.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:33.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T21:41:33.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:33.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:33 smithi067 ceph-mon[33346]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 859 objects/s recovering 2024-03-26T21:41:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:33 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:41:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:33 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:41:33.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[40587]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 859 objects/s recovering 2024-03-26T21:41:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:41:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:41:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[29406]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 activating+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 1 active+remapped+backfill_wait, 4 active+undersized+degraded+remapped+backfill_toofull; 5.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 50516/282168 objects degraded (17.903%); 40320/282168 objects misplaced (14.289%); 54 MiB/s, 859 objects/s recovering 2024-03-26T21:41:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:41:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:33 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:41:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.598 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:33.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:33.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:33.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:33.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:33.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:33.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.001 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:34.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:34.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:34.137 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:34.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:34.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:34.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:34.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:34.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:34.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:34 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:34.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:34 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:34.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:34.672 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:34.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:34 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:41:34.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:34.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:34.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:34.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:35.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:35.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:35.210 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:35.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:35.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:35.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:35.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:35.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[40587]: pgmap v572: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 954 op/s; 47652/269529 objects degraded (17.680%); 36605/269529 objects misplaced (13.581%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[29406]: pgmap v572: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 954 op/s; 47652/269529 objects degraded (17.680%); 36605/269529 objects misplaced (13.581%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:41:35.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:35 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:41:35.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:35 smithi067 ceph-mon[33346]: pgmap v572: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 954 op/s; 47652/269529 objects degraded (17.680%); 36605/269529 objects misplaced (13.581%); 32 MiB/s, 518 objects/s recovering 2024-03-26T21:41:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:35 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:41:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:35 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:41:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:35 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:41:35.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:35 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:41:35.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:35.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:35.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:35.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:36.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:36.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:36.157 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:36.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:36.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:36.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.546 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:36.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:36 smithi055 ceph-mon[29406]: 4.5 deep-scrub starts 2024-03-26T21:41:36.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:36 smithi055 ceph-mon[29406]: 4.5 deep-scrub ok 2024-03-26T21:41:36.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:36 smithi055 ceph-mon[40587]: 4.5 deep-scrub starts 2024-03-26T21:41:36.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:36 smithi055 ceph-mon[40587]: 4.5 deep-scrub ok 2024-03-26T21:41:36.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.675 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:36 smithi067 ceph-mon[33346]: 4.5 deep-scrub starts 2024-03-26T21:41:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:36 smithi067 ceph-mon[33346]: 4.5 deep-scrub ok 2024-03-26T21:41:36.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:36.815 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:36.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:36.942 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:36.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:37.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:37.068 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:37.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:37.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:37.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:37.348 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:37.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:37.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:37.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:37 smithi067 ceph-mon[33346]: pgmap v573: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 746 op/s; 47387/268734 objects degraded (17.633%); 36340/268734 objects misplaced (13.523%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:37 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:41:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:37 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:41:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 47387/268734 objects degraded (17.633%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[40587]: pgmap v573: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 746 op/s; 47387/268734 objects degraded (17.633%); 36340/268734 objects misplaced (13.523%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:41:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:41:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 47387/268734 objects degraded (17.633%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[29406]: pgmap v573: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 746 op/s; 47387/268734 objects degraded (17.633%); 36340/268734 objects misplaced (13.523%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:41:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:41:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 47387/268734 objects degraded (17.633%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:37.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:37.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.760 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:37.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:37.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:37.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:37.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:37.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:38.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:38.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:38.300 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:38.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:38.441 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:38.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:38.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:38.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:38.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:38.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:38.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:38.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:38.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:38.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:38.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:39.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:39.126 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:39.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:39.280 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:39.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:39.409 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:39.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:39.552 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[29406]: pgmap v574: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 1.03k op/s; 44777/259308 objects degraded (17.268%); 33541/259308 objects misplaced (12.935%); 35 MiB/s, 564 objects/s recovering 2024-03-26T21:41:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[29406]: 5.18 deep-scrub starts 2024-03-26T21:41:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[29406]: 5.18 deep-scrub ok 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[40587]: pgmap v574: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 1.03k op/s; 44777/259308 objects degraded (17.268%); 33541/259308 objects misplaced (12.935%); 35 MiB/s, 564 objects/s recovering 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[40587]: 5.18 deep-scrub starts 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[40587]: 5.18 deep-scrub ok 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:41:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:39 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:41:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:39 smithi067 ceph-mon[33346]: pgmap v574: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 1.03k op/s; 44777/259308 objects degraded (17.268%); 33541/259308 objects misplaced (12.935%); 35 MiB/s, 564 objects/s recovering 2024-03-26T21:41:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:39 smithi067 ceph-mon[33346]: 5.18 deep-scrub starts 2024-03-26T21:41:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:39 smithi067 ceph-mon[33346]: 5.18 deep-scrub ok 2024-03-26T21:41:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:39 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:41:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:39 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:41:39.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:39.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:39.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:39.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:39.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:39.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:40.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:40.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:40.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:40.257 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:40.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:40.400 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:40.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:40.539 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:40.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:40.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:40.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:40.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:40.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:40.951 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:40.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:41.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:41.191 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:41.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:41.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:41.367 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:41.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:41.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:41.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[29406]: pgmap v575: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 31 MiB/s, 498 objects/s recovering 2024-03-26T21:41:41.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:41:41.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:41:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[40587]: pgmap v575: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 31 MiB/s, 498 objects/s recovering 2024-03-26T21:41:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:41:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:41 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:41:41.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:41.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:41.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:41.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:41 smithi067 ceph-mon[33346]: pgmap v575: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 31 MiB/s, 498 objects/s recovering 2024-03-26T21:41:41.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:41 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:41:41.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:41 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:41:41.770 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:41.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:41.901 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:41.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:42.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:42.037 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:42.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:42.176 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:42.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:42.311 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:42.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:42.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:42.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:42.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 43937/255084 objects degraded (17.225%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:42.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 43937/255084 objects degraded (17.225%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:42.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:42.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:42.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 43937/255084 objects degraded (17.225%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:42.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:42.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:42.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:42.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:43.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:43.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:43.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:43.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:43.277 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:43.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:43.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:43.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:43 smithi055 ceph-mon[29406]: pgmap v576: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:41:43.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:43 smithi055 ceph-mon[40587]: pgmap v576: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:41:43.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:43.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:43 smithi067 ceph-mon[33346]: pgmap v576: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 43937/255084 objects degraded (17.225%); 32991/255084 objects misplaced (12.933%); 26 MiB/s, 417 objects/s recovering 2024-03-26T21:41:43.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:41:43.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:43.918 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:43.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:43.923 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:43.936 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:44.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.063 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:44.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:44.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:44.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:44.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:44.347 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:44 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:41:44.347 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:44 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:41:44.348 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:44 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:41:44.348 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:44 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:41:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.361 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:44.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:44.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:44.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:44.629 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:44.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:44 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:41:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:44 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:41:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:44.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:44.905 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:44.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:45.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:45.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:45.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.188 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:45.290 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:45.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:45.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:45.474 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:45.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:45.659 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[40587]: pgmap v577: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 40452/241578 objects degraded (16.745%); 29933/241578 objects misplaced (12.391%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:45.659 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[40587]: 4.9 deep-scrub starts 2024-03-26T21:41:45.659 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[40587]: 4.9 deep-scrub ok 2024-03-26T21:41:45.660 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[29406]: pgmap v577: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 40452/241578 objects degraded (16.745%); 29933/241578 objects misplaced (12.391%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:45.660 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[29406]: 4.9 deep-scrub starts 2024-03-26T21:41:45.660 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:45 smithi055 ceph-mon[29406]: 4.9 deep-scrub ok 2024-03-26T21:41:45.711 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:45 smithi067 ceph-mon[33346]: pgmap v577: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 40452/241578 objects degraded (16.745%); 29933/241578 objects misplaced (12.391%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:41:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:45 smithi067 ceph-mon[33346]: 4.9 deep-scrub starts 2024-03-26T21:41:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:45 smithi067 ceph-mon[33346]: 4.9 deep-scrub ok 2024-03-26T21:41:45.748 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:45.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:45.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:45.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:45.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:46.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.167 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:46.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:46.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:46.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:46.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:46.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:46.508 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[40587]: pgmap v578: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 40150/240672 objects degraded (16.682%); 29631/240672 objects misplaced (12.312%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:41:46.509 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:41:46.510 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:41:46.510 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:46 smithi055 ceph-mon[29406]: pgmap v578: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 40150/240672 objects degraded (16.682%); 29631/240672 objects misplaced (12.312%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:41:46.562 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:41:46.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:46.610 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:41:46.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:46.652 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:41:46.691 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:41:46.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:46.736 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:41:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:46 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:41:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:46 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:41:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:46 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:41:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:46 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:41:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:46 smithi067 ceph-mon[33346]: pgmap v578: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 801 op/s; 40150/240672 objects degraded (16.682%); 29631/240672 objects misplaced (12.312%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:41:46.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:46.777 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:41:46.818 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:41:46.823 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:41:46.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:46.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:46.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:47.020 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:47.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:47.155 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:47.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:47.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:47.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:47.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:41:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:41:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4256348615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 40150/240672 objects degraded (16.682%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:47.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:47.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:41:47.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:41:47.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4256348615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:47.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 40150/240672 objects degraded (16.682%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4256348615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 40150/240672 objects degraded (16.682%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:41:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:47.584 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:47.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:47.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:47.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:47.875 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:47.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:47.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:48.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:48.067 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:48.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:48.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:48.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:48.296 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:48.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:48.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:48.571 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:48.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:48.722 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:48 smithi067 ceph-mon[33346]: pgmap v579: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 37294/229950 objects degraded (16.218%); 26988/229950 objects misplaced (11.736%); 29 MiB/s, 471 objects/s recovering 2024-03-26T21:41:48.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:48 smithi055 ceph-mon[29406]: pgmap v579: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 37294/229950 objects degraded (16.218%); 26988/229950 objects misplaced (11.736%); 29 MiB/s, 471 objects/s recovering 2024-03-26T21:41:48.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:48 smithi055 ceph-mon[40587]: pgmap v579: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 37294/229950 objects degraded (16.218%); 26988/229950 objects misplaced (11.736%); 29 MiB/s, 471 objects/s recovering 2024-03-26T21:41:48.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:48.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:48.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:48.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:49.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:49.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:49.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:49.296 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:49.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:49.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:49.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:49.571 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:49.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:49.699 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:49.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:41:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:49 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:41:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:49 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:41:49.781 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:49 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:41:49.782 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:49 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:41:49.782 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:49 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:41:49.782 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:49 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:41:49.784 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:41:49.785 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:41:49.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:49.846 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:49.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:49.981 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:49.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:50.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:50.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:50.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:50.382 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:50.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:50.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:50.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:50 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:41:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:50 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:41:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:50 smithi067 ceph-mon[33346]: pgmap v580: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 20 MiB/s, 313 objects/s recovering 2024-03-26T21:41:50.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:41:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:41:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[40587]: pgmap v580: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 20 MiB/s, 313 objects/s recovering 2024-03-26T21:41:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:41:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:41:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:50 smithi055 ceph-mon[29406]: pgmap v580: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 20 MiB/s, 313 objects/s recovering 2024-03-26T21:41:50.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:50.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:51.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:51.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:51.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:51.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:51.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:51.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:51.617 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:51.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:51.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:51.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:51.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:51.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:51.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.024 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:52.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:52.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:52.169 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:52.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:52.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:52.408 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:52.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:41:52.494 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:52.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:52.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: 4.d deep-scrub starts 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: 4.d deep-scrub ok 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: pgmap v582: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 36323/224985 objects degraded (16.145%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[40587]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T21:41:52.691 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: 4.d deep-scrub starts 2024-03-26T21:41:52.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: 4.d deep-scrub ok 2024-03-26T21:41:52.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T21:41:52.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: pgmap v582: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:41:52.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 36323/224985 objects degraded (16.145%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:52.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:52 smithi055 ceph-mon[29406]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T21:41:52.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:52.743 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: 4.d deep-scrub starts 2024-03-26T21:41:52.743 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: 4.d deep-scrub ok 2024-03-26T21:41:52.743 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T21:41:52.743 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: pgmap v582: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 36323/224985 objects degraded (16.145%); 26339/224985 objects misplaced (11.707%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:41:52.744 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 36323/224985 objects degraded (16.145%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:52.744 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:52 smithi067 ceph-mon[33346]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T21:41:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:52.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:52.834 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:41:52.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:52.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:52.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:53.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:53.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:53.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:53.230 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:53.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:53.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:53.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:53.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:53.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:53.657 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:53.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:53.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:53.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:53.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:53.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:54.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:54.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:54.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.222 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:54.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:54.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:54.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:54.496 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:54.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:54.664 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T21:41:54.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:54 smithi067 ceph-mon[33346]: pgmap v584: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 32905/210720 objects degraded (15.616%); 23674/210720 objects misplaced (11.235%); 27 MiB/s, 436 objects/s recovering 2024-03-26T21:41:54.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:54 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:41:54.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:54 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:41:54.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:54 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/746791446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:54.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:54 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:54.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:54.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:54.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:54.912 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:54.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:55.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[40587]: pgmap v584: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 32905/210720 objects degraded (15.616%); 23674/210720 objects misplaced (11.235%); 27 MiB/s, 436 objects/s recovering 2024-03-26T21:41:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:41:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:41:55.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/746791446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[29406]: pgmap v584: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 32905/210720 objects degraded (15.616%); 23674/210720 objects misplaced (11.235%); 27 MiB/s, 436 objects/s recovering 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/746791446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:55.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:54 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:41:55.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:55.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:55.213 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:41:55.213 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:41:55.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:55.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:55.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:55.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:55.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:55.724 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:55 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:41:55.724 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:55 smithi067 ceph-mon[33346]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T21:41:55.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:55.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:55.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:55.909 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:55.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:55.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:55 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:41:55.933 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:55 smithi055 ceph-mon[40587]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T21:41:55.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:55 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:41:55.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:55 smithi055 ceph-mon[29406]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T21:41:55.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:41:55.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T21:41:56.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:56.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:56.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:56.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:56.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:41:56.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:56.505 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:56.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:56.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:56.659 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:56.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:56.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:56.872 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:56 smithi055 ceph-mon[29406]: pgmap v586: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 32564/209697 objects degraded (15.529%); 23333/209697 objects misplaced (11.127%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:41:56.872 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:56 smithi055 ceph-mon[40587]: pgmap v586: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 32564/209697 objects degraded (15.529%); 23333/209697 objects misplaced (11.127%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:41:56.874 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:56 smithi067 ceph-mon[33346]: pgmap v586: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 135 active+clean, 5 active+undersized+degraded+remapped+backfill_toofull; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 32564/209697 objects degraded (15.529%); 23333/209697 objects misplaced (11.127%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:41:56.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:56.930 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:56.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:56.982 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:41:57.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:57.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:57.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:57.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:57.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:41:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:41:57.470 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:57.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:41:57.504 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:57.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:57.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:57.724 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:41:57.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.787 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:57.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:57.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 32564/209697 objects degraded (15.529%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/803495592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:57.921 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:57.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:57.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:58.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:41:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:41:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T21:41:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 32564/209697 objects degraded (15.529%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:58.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/803495592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 32564/209697 objects degraded (15.529%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/803495592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:41:58.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:41:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:58.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:58.216 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:58.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:58.345 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:58.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:41:58.484 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:41:58.577 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:41:58.577 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:41:58.586 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:58.609 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:58.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:58.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:58.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:41:58.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:58.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:41:58.972 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: 6.e deep-scrub starts 2024-03-26T21:41:58.972 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: 6.e deep-scrub ok 2024-03-26T21:41:58.973 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: pgmap v588: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 4 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.45k op/s; 29948/198834 objects degraded (15.062%); 21456/198834 objects misplaced (10.791%); 30 MiB/s, 480 objects/s recovering 2024-03-26T21:41:58.973 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:58.973 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:41:58.973 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:58 smithi067 ceph-mon[33346]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: 6.e deep-scrub starts 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: 6.e deep-scrub ok 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: pgmap v588: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 4 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.45k op/s; 29948/198834 objects degraded (15.062%); 21456/198834 objects misplaced (10.791%); 30 MiB/s, 480 objects/s recovering 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:41:58.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[40587]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: 6.e deep-scrub starts 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: 6.e deep-scrub ok 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: pgmap v588: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 4 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.45k op/s; 29948/198834 objects degraded (15.062%); 21456/198834 objects misplaced (10.791%); 30 MiB/s, 480 objects/s recovering 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:41:58.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-mon[29406]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T21:41:58.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:59.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:41:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:41:59.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:41:59.317 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:41:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:41:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:41:59.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:41:59.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:41:59.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.511 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:59.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:59.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:41:59.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:41:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:41:59.727 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:41:59.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:41:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:41:59.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:41:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:41:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:41:59.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:41:59.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:41:59 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:00.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:00.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:41:59 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:00.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:41:59 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:42:00.087 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:00.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:00.225 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:00.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:00.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:00.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:00.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.498 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:00.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:00.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:00.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:00.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:00.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:00.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:00.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:00.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:00.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:00 smithi067 ceph-mon[33346]: pgmap v590: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:42:00.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:00.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:00.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:00 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:01.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:01.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:01.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[40587]: pgmap v590: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[29406]: pgmap v590: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 929 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:42:01.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:00 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:01.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.193 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:01.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:01.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:01.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:01.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:01.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:01.624 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:01.642 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:01.642 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:01.726 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:01.763 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:01.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:01.906 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:01.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:02.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.049 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:02.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:02.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:02.173 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:02.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:02.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:42:02.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.485 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:02.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:02.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:02.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:02.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:02.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:02.875 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:02.917 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:02.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:02 smithi067 ceph-mon[33346]: pgmap v591: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 755 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 11 MiB/s, 176 objects/s recovering 2024-03-26T21:42:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 28972/194004 objects degraded (14.934%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:02 smithi067 ceph-mon[33346]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T21:42:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:02.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[40587]: pgmap v591: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 755 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 11 MiB/s, 176 objects/s recovering 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 28972/194004 objects degraded (14.934%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[40587]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:03.014 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[29406]: pgmap v591: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 738 B/s rd, 0 B/s wr, 755 op/s; 28972/194004 objects degraded (14.934%); 20832/194004 objects misplaced (10.738%); 11 MiB/s, 176 objects/s recovering 2024-03-26T21:42:03.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 28972/194004 objects degraded (14.934%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:03.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[29406]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T21:42:03.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:42:03.015 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:03.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:03.066 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:03.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:03.202 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:03.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.342 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:03.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:03.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:03.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:03.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:03.627 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:03.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:03.770 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:03.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:03.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:03.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:04.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:04.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:04.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:04.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:04.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:04.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:04.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:04.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:04.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:04.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:04.799 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:04.799 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:04.833 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:42:04.878 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:42:04.879 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:04 smithi055 ceph-mon[40587]: pgmap v593: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 25990/180438 objects degraded (14.404%); 18734/180438 objects misplaced (10.383%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:42:04.879 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:04 smithi055 ceph-mon[29406]: pgmap v593: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 25990/180438 objects degraded (14.404%); 18734/180438 objects misplaced (10.383%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:42:04.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:04.922 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:42:04.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:04.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:04.962 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:42:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:04 smithi067 ceph-mon[33346]: pgmap v593: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 25990/180438 objects degraded (14.404%); 18734/180438 objects misplaced (10.383%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:42:04.999 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:42:05.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:05.037 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:42:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:05.075 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:42:05.108 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:42:05.110 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:42:05.111 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:42:05.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:05.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:05.321 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:05.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:05.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:05.580 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:05.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:42:05.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:05.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:05.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:05 smithi055 ceph-mon[40587]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T21:42:05.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:05 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1951970490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:05.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:05 smithi055 ceph-mon[29406]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T21:42:05.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:05 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1951970490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:05.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:05.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:05.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:05.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:05.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:05 smithi067 ceph-mon[33346]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T21:42:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:05 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1951970490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:05.993 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:06.032 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:42:06.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.146 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:06.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:06.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:06.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:06.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.422 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:06.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:06.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:06.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:06.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:06.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:06.816 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T21:42:06.894 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:06.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:06.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:07.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[29406]: pgmap v595: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 820 op/s; 25681/179511 objects degraded (14.306%); 18425/179511 objects misplaced (10.264%); 9.6 MiB/s, 152 objects/s recovering 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[29406]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/871256250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 25681/179511 objects degraded (14.306%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[40587]: pgmap v595: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 820 op/s; 25681/179511 objects degraded (14.306%); 18425/179511 objects misplaced (10.264%); 9.6 MiB/s, 152 objects/s recovering 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[40587]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/871256250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:06 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 25681/179511 objects degraded (14.306%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:07.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:06 smithi067 ceph-mon[33346]: pgmap v595: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 136 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 820 op/s; 25681/179511 objects degraded (14.306%); 18425/179511 objects misplaced (10.264%); 9.6 MiB/s, 152 objects/s recovering 2024-03-26T21:42:07.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:06 smithi067 ceph-mon[33346]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T21:42:07.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/871256250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:06 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:42:07.096 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:06 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 25681/179511 objects degraded (14.306%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:07.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:07.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:07.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:07.422 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:07.467 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:07.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.625 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:07.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:07.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T21:42:07.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.763 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:07.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:07.802 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:07.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:07.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:07.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:07.946 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:42:07.946 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[29406]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T21:42:07.946 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:07.946 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:42:07.947 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[40587]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T21:42:07.947 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:08.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.051 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:08.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:08.081 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:07 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:42:08.081 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:07 smithi067 ceph-mon[33346]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T21:42:08.081 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:08.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:08.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:42:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:42:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:42:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:42:08.188 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:42:08.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:08.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.344 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:08.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:08.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:08.485 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:08.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:08.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:08.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:42:08.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:08.789 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:08.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:08.930 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:08.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:09.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:09.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:42:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[29406]: pgmap v598: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.10k op/s; 23192/170112 objects degraded (13.633%); 16696/170112 objects misplaced (9.815%); 25 MiB/s, 394 objects/s recovering 2024-03-26T21:42:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:09.068 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:09.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[40587]: pgmap v598: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.10k op/s; 23192/170112 objects degraded (13.633%); 16696/170112 objects misplaced (9.815%); 25 MiB/s, 394 objects/s recovering 2024-03-26T21:42:09.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:09.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:09.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:08 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:09.080 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:09.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:08 smithi067 ceph-mon[33346]: pgmap v598: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 1.10k op/s; 23192/170112 objects degraded (13.633%); 16696/170112 objects misplaced (9.815%); 25 MiB/s, 394 objects/s recovering 2024-03-26T21:42:09.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:08 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:09.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:08 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:09.083 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:08 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:09.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:09.213 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:09.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:09.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:09.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:09.452 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:09.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:09.540 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:09.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:09.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:09.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:09.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:09.901 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:09 smithi067 ceph-mon[33346]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T21:42:09.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:09.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:10.025 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:10.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:10.071 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:09 smithi055 ceph-mon[29406]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T21:42:10.072 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:09 smithi055 ceph-mon[40587]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T21:42:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:10.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:10.248 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:10.350 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:10.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:10.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:10.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:10.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:10.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:10.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:10.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:10.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:10.937 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:10.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:11.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:11.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:11.116 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:10 smithi067 ceph-mon[33346]: pgmap v600: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 29 MiB/s, 469 objects/s recovering 2024-03-26T21:42:11.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:11.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:10 smithi055 ceph-mon[29406]: pgmap v600: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 29 MiB/s, 469 objects/s recovering 2024-03-26T21:42:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:10 smithi055 ceph-mon[40587]: pgmap v600: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 29 MiB/s, 469 objects/s recovering 2024-03-26T21:42:11.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:11.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:11.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:11.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:11.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:11.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:11.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:11.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:11.911 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:11.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:11.950 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 21716/165711 objects degraded (13.105%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:11.950 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 21716/165711 objects degraded (13.105%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:12.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:12.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:12.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:12.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:12.187 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 21716/165711 objects degraded (13.105%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:12.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:12.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:12.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:12.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:12.551 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:12.593 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:12.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:12.718 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:12.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:12.846 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:12.890 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:42:12.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:12.991 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:12.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:13.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:13.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:12 smithi067 ceph-mon[33346]: pgmap v601: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 767 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:42:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:13.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84024 8.5 8:a9a6864a:::benchmark_data_smithi055_110126_object84023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 369.83801405299999, 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18444987099999999, 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84024 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531495+0000", 2024-03-26T21:42:13.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531496+0000", 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.73e-06 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531543+0000", 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6681999999999998e-05 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531556+0000", 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3032e-05 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.531564+0000", 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1289999999999996e-06 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555104+0000", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023539966999999998 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555293+0000", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018880700000000001 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.555499+0000", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020556200000000001 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564809+0000", 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093100309999999999 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567429+0000", 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002620645 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712266+0000", 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14483680500000001 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.712311+0000", 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4746000000000002e-05 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.715945+0000", 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036337349999999999 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.286 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84036 8.5 8:a733162b:::benchmark_data_smithi055_110126_object84035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 369.82506259000002, 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18335849000000001, 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84036 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544446+0000", 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544447+0000", 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.78e-07 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544508+0000", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0568e-05 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544530+0000", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2268000000000001e-05 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544536+0000", 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2829999999999998e-06 2024-03-26T21:42:13.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.564845+0000", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020308641999999998 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565017+0000", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017215199999999999 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.565220+0000", 2024-03-26T21:42:13.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020294399999999999 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576111+0000", 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010890982 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576274+0000", 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000162819 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724342+0000", 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14806820500000001 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.724408+0000", 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6094999999999997e-05 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727805+0000", 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033966539999999998 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.292 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84038 8.5 8:a3a8abbc:::benchmark_data_smithi055_110126_object84037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 369.82496387200001, 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18689033499999999, 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84038 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544545+0000", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.06e-07 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544585+0000", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9360000000000003e-05 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544592+0000", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-06 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544598+0000", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5310000000000001e-06 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567532+0000", 2024-03-26T21:42:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022934059 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567708+0000", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017579499999999999 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.567936+0000", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000228173 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576172+0000", 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082357030000000005 2024-03-26T21:42:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576281+0000", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109747 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727868+0000", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.151586689 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.727982+0000", 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011343700000000001 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731435+0000", 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003453631 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84039 8.5 8:a8845c01:::benchmark_data_smithi055_110126_object84038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 369.82490208000002, 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.190558218, 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84039 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:42:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544607+0000", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4400000000000001e-07 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544631+0000", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.442e-05 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544638+0000", 2024-03-26T21:42:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3790000000000003e-06 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.544643+0000", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7160000000000002e-06 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570111+0000", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025468094 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570292+0000", 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018180200000000001 2024-03-26T21:42:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.570501+0000", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208727 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.576257+0000", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057555239999999997 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584951+0000", 2024-03-26T21:42:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086946679999999991 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731584+0000", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.146632868 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.731658+0000", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3287999999999994e-05 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735165+0000", 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035072879999999999 2024-03-26T21:42:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:84048 8.5 8:a98c3a1d:::benchmark_data_smithi055_110126_object84047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 369.813436975, 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.182857732, 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 84048 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556072+0000", 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556073+0000", 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.584e-06 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556178+0000", 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104714 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556189+0000", 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0582e-05 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.556196+0000", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8259999999999993e-06 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573081+0000", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016884952000000002 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573261+0000", 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017950600000000001 2024-03-26T21:42:13.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.573469+0000", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020786200000000001 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.584956+0000", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011487264000000001 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.585011+0000", 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4629000000000002e-05 2024-03-26T21:42:13.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735223+0000", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.150211927 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.735322+0000", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9017000000000004e-05 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:02.738929+0000", 2024-03-26T21:42:13.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036078690000000001 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:86639 8.5 8:a036fec7:::benchmark_data_smithi055_110126_object86638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 366.33993619199998, 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.236906851, 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 86639 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029572+0000", 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029574+0000", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.271e-06 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029628+0000", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4159000000000001e-05 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029639+0000", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0638e-05 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.029646+0000", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1629999999999999e-06 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.036918+0000", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072720809999999997 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037144+0000", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022611500000000001 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.037343+0000", 2024-03-26T21:42:13.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019870900000000001 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.040140+0000", 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00279715 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.042107+0000", 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019675669999999999 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262797+0000", 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.220689684 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.262862+0000", 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5030000000000006e-05 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:06.266479+0000", 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036172840000000001 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108533 8.5 8:a9c0f906:::benchmark_data_smithi055_110126_object108532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 341.11681661300003, 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21060493199999999, 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108533 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252692+0000", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252694+0000", 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5960000000000001e-06 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252736+0000", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2651999999999998e-05 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252750+0000", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3966e-05 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.252759+0000", 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-03-26T21:42:13.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271069+0000", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018310448999999999 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271232+0000", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016281499999999999 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.271366+0000", 2024-03-26T21:42:13.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013423299999999999 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275699+0000", 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043322029999999997 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369423+0000", 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093724093999999994 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.460020+0000", 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090597103999999998 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.460100+0000", 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0211999999999997e-05 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463297+0000", 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031969149999999998 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108536 8.5 8:adbf2f7d:::benchmark_data_smithi055_110126_object108535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 341.114209362, 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21185876100000001, 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108536 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.322 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255299+0000", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255300+0000", 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4900000000000005e-07 2024-03-26T21:42:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255347+0000", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6761000000000002e-05 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255357+0000", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0264e-05 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255364+0000", 2024-03-26T21:42:13.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7000000000000002e-06 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273327+0000", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017963257999999999 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273484+0000", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015665699999999999 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.273673+0000", 2024-03-26T21:42:13.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189648 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.277642+0000", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039683460000000002 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369476+0000", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091834425999999997 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463380+0000", 2024-03-26T21:42:13.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093904038999999995 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.463452+0000", 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1627999999999999e-05 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467158+0000", 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037061849999999999 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15096.0:108537 8.5 8:a9ba36a4:::benchmark_data_smithi055_110126_object108536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 341.11405654399999, 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21534863500000001, 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15096", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/243374766", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 108537 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255452+0000", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255458+0000", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1469999999999998e-06 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255515+0000", 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6891000000000001e-05 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255524+0000", 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5019999999999992e-06 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.255529+0000", 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.186e-06 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275731+0000", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02020183 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275839+0000", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010788 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.275964+0000", 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012569299999999999 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.277748+0000", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001783506 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.369518+0000", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091770298 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467221+0000", 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097703264999999997 2024-03-26T21:42:13.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.467338+0000", 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116978 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:36:31.470801+0000", 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034624590000000002 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147853 9.1 9:8b320b50:::benchmark_data_smithi055_135863_object17202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.659982834999994, 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.068236535, 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147853 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709527+0000", 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3900000000000003e-07 2024-03-26T21:42:13.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709530+0000", 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3950000000000001e-06 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709538+0000", 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4610000000000002e-06 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709543+0000", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2839999999999997e-06 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709581+0000", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8556999999999997e-05 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709587+0000", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5509999999999999e-06 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665317+0000", 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95572950800000001 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665332+0000", 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5940999999999999e-05 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.770155+0000", 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.104822763 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.771678+0000", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001522949 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.771834+0000", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015530700000000001 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773789+0000", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019550790000000002 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775042+0000", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001252925 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777663+0000", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002621717 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777700+0000", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6646000000000001e-05 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777762+0000", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2513000000000003e-05 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147865 9.1 9:83cb768e:::benchmark_data_smithi055_135863_object17214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.639700426000005, 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0480903599999998, 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.341 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147865 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729810+0000", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3209999999999999e-06 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729814+0000", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6330000000000004e-06 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729826+0000", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2021000000000001e-05 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729833+0000", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8650000000000003e-06 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729879+0000", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6397000000000003e-05 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729886+0000", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4540000000000004e-06 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665361+0000", 2024-03-26T21:42:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93547465600000002 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665369+0000", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0639999999999994e-06 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773836+0000", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.108466988 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773917+0000", 2024-03-26T21:42:13.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1817999999999995e-05 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773990+0000", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2677999999999998e-05 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775202+0000", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001211424 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775237+0000", 2024-03-26T21:42:13.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4974000000000001e-05 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777796+0000", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025594179999999999 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777832+0000", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6492999999999998e-05 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777899+0000", 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6155999999999995e-05 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.347 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166763 9.1 9:8eba03ed:::benchmark_data_smithi055_135863_object36112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.787357073999999, 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.149474361, 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166763 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582153+0000", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4270000000000001e-06 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582159+0000", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.049e-06 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582175+0000", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5739999999999998e-05 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582184+0000", 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.772e-06 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582242+0000", 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8118999999999997e-05 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582410+0000", 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167897 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582557+0000", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014689600000000001 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.610257+0000", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027700675000000001 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.610271+0000", 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4199000000000001e-05 2024-03-26T21:42:13.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731535+0000", 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1212639820000001 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731626+0000", 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0605000000000004e-05 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2908 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94208173341440 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-03-26T21:42:13.353 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.044305+0000", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.325203915000003, 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56165444600000003, 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2908 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044305+0000", 2024-03-26T21:42:13.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044305+0000", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044306+0000", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5260000000000001e-06 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044312+0000", 2024-03-26T21:42:13.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.964e-06 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044331+0000", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9208999999999999e-05 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044343+0000", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1246e-05 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044463+0000", 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120133 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044477+0000", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3818999999999999e-05 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605611+0000", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56113398000000003 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605814+0000", 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020320900000000001 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605959+0000", 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014536 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2910 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94208178013312 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.044415+0000", 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.325093787999997, 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56564689499999998, 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2910 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044415+0000", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044415+0000", 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044415+0000", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6499999999999999e-07 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044419+0000", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3809999999999999e-06 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044426+0000", 2024-03-26T21:42:13.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7349999999999999e-06 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044432+0000", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3690000000000003e-06 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044502+0000", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9918000000000004e-05 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044513+0000", 2024-03-26T21:42:13.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1087000000000001e-05 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.609804+0000", 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56529069099999996 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.609962+0000", 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015883800000000001 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.610062+0000", 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9310999999999998e-05 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2913 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94208173341440 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.044968+0000", 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.32454019, 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.58203009999999999, 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:42:13.363 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2913 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044968+0000", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044968+0000", 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044969+0000", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0800000000000004e-07 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044973+0000", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0400000000000003e-06 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044982+0000", 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.473e-06 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.044989+0000", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2799999999999998e-06 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045047+0000", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8418999999999998e-05 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045058+0000", 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0533999999999999e-05 2024-03-26T21:42:13.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605992+0000", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56093457199999996 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.606074+0000", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1867999999999996e-05 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.626999+0000", 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020924306 2024-03-26T21:42:13.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2915 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94208178013312 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.045002+0000", 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.324506329999998, 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.58231841799999995, 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2915 2024-03-26T21:42:13.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045002+0000", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045002+0000", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045003+0000", 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8800000000000003e-07 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045006+0000", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7310000000000002e-06 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045012+0000", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2210000000000002e-06 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045018+0000", 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3749999999999999e-06 2024-03-26T21:42:13.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045064+0000", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6054999999999999e-05 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.045071+0000", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7839999999999998e-06 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.610088+0000", 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.56501741100000002 2024-03-26T21:42:13.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.610163+0000", 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5078000000000002e-05 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.627321+0000", 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017157275 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166872 9.1 9:8abb16dd:::benchmark_data_smithi055_135863_object36221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T21:42:13.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:26.820502+0000", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.549006648999999, 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.250211085, 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166872 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820502+0000", 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820502+0000", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820504+0000", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6270000000000001e-06 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820510+0000", 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4400000000000002e-06 2024-03-26T21:42:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820528+0000", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8063e-05 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820539+0000", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0915e-05 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820618+0000", 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8650000000000001e-05 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820772+0000", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015451899999999999 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4,7", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.820890+0000", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011791 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826786+0000", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058954059999999997 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826837+0000", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1338999999999998e-05 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826851+0000", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4068e-05 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.070406+0000", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.24355512600000001 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.070574+0000", 2024-03-26T21:42:13.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016783799999999999 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.070713+0000", 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013918399999999999 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166874 9.1 9:857252d8:::benchmark_data_smithi055_135863_object36223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:26.822380+0000", 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.547129009999999, 2024-03-26T21:42:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.24867329799999999, 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166874 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822380+0000", 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822380+0000", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822382+0000", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9379999999999999e-06 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822387+0000", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0039999999999999e-06 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822399+0000", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2602000000000001e-05 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.822408+0000", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3489999999999992e-06 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823716+0000", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013083509999999999 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823860+0000", 2024-03-26T21:42:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144155 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4,7", 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823945+0000", 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4929e-05 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826888+0000", 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00294342 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826932+0000", 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3878000000000002e-05 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.830008+0000", 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003075445 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.070766+0000", 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.24075780399999999 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.070910+0000", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014450899999999999 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071053+0000", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291400000000001 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166877 9.1 9:8cfefe97:::benchmark_data_smithi055_135863_object36226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:26.823372+0000", 2024-03-26T21:42:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.546136928999999, 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.247939206, 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166877 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823372+0000", 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823372+0000", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823373+0000", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.232e-06 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823379+0000", 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7200000000000003e-06 2024-03-26T21:42:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823394+0000", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.502e-05 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.823412+0000", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8465e-05 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.826966+0000", 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035541610000000001 2024-03-26T21:42:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.827100+0000", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013392800000000001 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4,7", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.827240+0000", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139393 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.830179+0000", 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029390839999999998 2024-03-26T21:42:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.830211+0000", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2543999999999999e-05 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.830898+0000", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068636800000000003 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071100+0000", 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.24020274899999999 2024-03-26T21:42:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071191+0000", 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0184000000000003e-05 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071311+0000", 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120358 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166880 9.1 9:8f3d9150:::benchmark_data_smithi055_135863_object36229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:26.832575+0000", 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.536933300000001, 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.238979365, 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166880 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832575+0000", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832575+0000", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832577+0000", 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.198e-06 2024-03-26T21:42:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832581+0000", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1420000000000003e-06 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832592+0000", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0781999999999999e-05 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832597+0000", 2024-03-26T21:42:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.913e-06 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832653+0000", 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5841999999999998e-05 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832803+0000", 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015020999999999999 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4,7", 2024-03-26T21:42:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.832962+0000", 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000158419 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.835625+0000", 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026634900000000001 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.835674+0000", 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8863999999999998e-05 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.835765+0000", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0470000000000006e-05 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071358+0000", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.23559356100000001 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071431+0000", 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2538000000000002e-05 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:27.071555+0000", 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123936 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:13.397 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:42:13.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:13.398 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:12 smithi055 ceph-mon[40587]: pgmap v601: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 767 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:42:13.398 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:13.399 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:12 smithi055 ceph-mon[29406]: pgmap v601: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 767 op/s; 21716/165711 objects degraded (13.105%); 16128/165711 objects misplaced (9.733%); 24 MiB/s, 376 objects/s recovering 2024-03-26T21:42:13.399 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:13.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:13.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:13.565 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:13.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:13.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:13.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:13.912 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T21:42:13.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:13.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:13.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:14.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:14.100 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:14.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:14.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:13 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: osdmap e293: 8 total, 8 up, 7 in 2024-03-26T21:42:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:42:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[40587]: osdmap e293: 8 total, 8 up, 7 in 2024-03-26T21:42:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:42:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/147042366' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[29406]: osdmap e293: 8 total, 8 up, 7 in 2024-03-26T21:42:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:42:14.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:14.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:14.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:14.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:14.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:14.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:14.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:14.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:14.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:14.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:15.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:15.038 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:15.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:15.169 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: pgmap v603: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 18606/151611 objects degraded (12.272%); 14184/151611 objects misplaced (9.356%); 39 MiB/s, 617 objects/s recovering 2024-03-26T21:42:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:42:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:14 smithi067 ceph-mon[33346]: osdmap e294: 8 total, 8 up, 7 in 2024-03-26T21:42:15.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:15.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[40587]: pgmap v603: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 18606/151611 objects degraded (12.272%); 14184/151611 objects misplaced (9.356%); 39 MiB/s, 617 objects/s recovering 2024-03-26T21:42:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:42:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[40587]: osdmap e294: 8 total, 8 up, 7 in 2024-03-26T21:42:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[29406]: pgmap v603: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s; 18606/151611 objects degraded (12.272%); 14184/151611 objects misplaced (9.356%); 39 MiB/s, 617 objects/s recovering 2024-03-26T21:42:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3902480911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:42:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:14 smithi055 ceph-mon[29406]: osdmap e294: 8 total, 8 up, 7 in 2024-03-26T21:42:15.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:15.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:15.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:15.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:15.664 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:15.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:15.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:15.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:15.931 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:15.966 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:15.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:16.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:16.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:16.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:15 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:15 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:16.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:15 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:16.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:15 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:16.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:15 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:16.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:15 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:16.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:16.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:16.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:16.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:16.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:16.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:16.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:16.772 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:16.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:16.908 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:16.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:17.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:17.033 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:17.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:17.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:17.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:16 smithi067 ceph-mon[33346]: pgmap v605: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 739 op/s; 18324/150765 objects degraded (12.154%); 13902/150765 objects misplaced (9.221%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:42:17.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:16 smithi067 ceph-mon[33346]: osdmap e295: 8 total, 8 up, 7 in 2024-03-26T21:42:17.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:16 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 18324/150765 objects degraded (12.154%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:17.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:16 smithi067 ceph-mon[33346]: osdmap e296: 8 total, 8 up, 7 in 2024-03-26T21:42:17.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[40587]: pgmap v605: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 739 op/s; 18324/150765 objects degraded (12.154%); 13902/150765 objects misplaced (9.221%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:42:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[40587]: osdmap e295: 8 total, 8 up, 7 in 2024-03-26T21:42:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 18324/150765 objects degraded (12.154%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[40587]: osdmap e296: 8 total, 8 up, 7 in 2024-03-26T21:42:17.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[29406]: pgmap v605: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 759 B/s rd, 0 B/s wr, 739 op/s; 18324/150765 objects degraded (12.154%); 13902/150765 objects misplaced (9.221%); 21 MiB/s, 338 objects/s recovering 2024-03-26T21:42:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[29406]: osdmap e295: 8 total, 8 up, 7 in 2024-03-26T21:42:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 18324/150765 objects degraded (12.154%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:16 smithi055 ceph-mon[29406]: osdmap e296: 8 total, 8 up, 7 in 2024-03-26T21:42:17.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:17.418 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:17.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:17.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:17.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:17.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:17.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:17.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:17.807 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:17.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:17.946 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:17.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:18.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:18.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:18.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:18.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[40587]: osdmap e297: 8 total, 8 up, 7 in 2024-03-26T21:42:18.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:18.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:18.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:17 smithi055 ceph-mon[29406]: osdmap e297: 8 total, 8 up, 7 in 2024-03-26T21:42:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:18.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:18.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:17 smithi067 ceph-mon[33346]: osdmap e297: 8 total, 8 up, 7 in 2024-03-26T21:42:18.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:18.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:18.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:18.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:18.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.589 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:18.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:18.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:18.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:18.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:18.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:18.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:18.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:18.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:19.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:19.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:19.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:18 smithi067 ceph-mon[33346]: pgmap v608: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 807 op/s; 16867/143778 objects degraded (11.731%); 12615/143778 objects misplaced (8.774%); 25 MiB/s, 407 objects/s recovering 2024-03-26T21:42:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:18 smithi067 ceph-mon[33346]: osdmap e298: 8 total, 8 up, 7 in 2024-03-26T21:42:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:18 smithi055 ceph-mon[40587]: pgmap v608: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 807 op/s; 16867/143778 objects degraded (11.731%); 12615/143778 objects misplaced (8.774%); 25 MiB/s, 407 objects/s recovering 2024-03-26T21:42:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:18 smithi055 ceph-mon[40587]: osdmap e298: 8 total, 8 up, 7 in 2024-03-26T21:42:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:18 smithi055 ceph-mon[29406]: pgmap v608: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 137 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 807 op/s; 16867/143778 objects degraded (11.731%); 12615/143778 objects misplaced (8.774%); 25 MiB/s, 407 objects/s recovering 2024-03-26T21:42:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:18 smithi055 ceph-mon[29406]: osdmap e298: 8 total, 8 up, 7 in 2024-03-26T21:42:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:19.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:19.400 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:19.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:19.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.683 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:19.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:19.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:19.817 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:19.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:19.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:20.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:20.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:20 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:20.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:20 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:20.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.230 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:20.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:20.330 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:20.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:20.433 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:20 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:20.477 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:20.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:42:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:20.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:20.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:20.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:20.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:20.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:20.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:20.910 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:20.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:20.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:20.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T21:42:21.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:21.074 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:21.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:21.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:21 smithi055 ceph-mon[29406]: pgmap v611: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 861 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 46 MiB/s, 731 objects/s recovering 2024-03-26T21:42:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:21 smithi055 ceph-mon[40587]: pgmap v611: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 861 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 46 MiB/s, 731 objects/s recovering 2024-03-26T21:42:21.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:21.322 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:21 smithi067 ceph-mon[33346]: pgmap v611: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 861 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 46 MiB/s, 731 objects/s recovering 2024-03-26T21:42:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:21.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:21.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:21.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:21.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:21.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:21.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:42:21.847 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:21.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:21.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:22.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:22.061 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:22.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:22.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:22.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:22.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 14997/140442 objects degraded (10.678%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 14997/140442 objects degraded (10.678%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:22.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:22.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:22.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 14997/140442 objects degraded (10.678%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T21:42:22.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:22.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:22.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:22.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:22.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:23.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:23.100 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:23 smithi067 ceph-mon[33346]: pgmap v612: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 905 B/s rd, 0 B/s wr, 609 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 32 MiB/s, 517 objects/s recovering 2024-03-26T21:42:23.100 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/563812788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:23.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:23 smithi055 ceph-mon[29406]: pgmap v612: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 905 B/s rd, 0 B/s wr, 609 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 32 MiB/s, 517 objects/s recovering 2024-03-26T21:42:23.143 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/563812788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:23.143 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:23 smithi055 ceph-mon[40587]: pgmap v612: 148 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 138 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 905 B/s rd, 0 B/s wr, 609 op/s; 14997/140442 objects degraded (10.678%); 12266/140442 objects misplaced (8.734%); 32 MiB/s, 517 objects/s recovering 2024-03-26T21:42:23.143 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/563812788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:23.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:42:23.268 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:42:23.269 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:42:23.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:23.308 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:23.308 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:23.317 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:23.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:23.558 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:23.592 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:23.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:23.728 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:23.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:23.856 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:23.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:23.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:23.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:24.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:42:24.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:24.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:24.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:24.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:24.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:24.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:24.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:24.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:24.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:24.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:24.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:24.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:24.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:25.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:25.126 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:25.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:25 smithi067 ceph-mon[33346]: pgmap v613: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 12193/128685 objects degraded (9.475%); 11455/128685 objects misplaced (8.902%); 44 MiB/s, 707 objects/s recovering 2024-03-26T21:42:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:25.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:25 smithi055 ceph-mon[29406]: pgmap v613: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 12193/128685 objects degraded (9.475%); 11455/128685 objects misplaced (8.902%); 44 MiB/s, 707 objects/s recovering 2024-03-26T21:42:25.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:25 smithi055 ceph-mon[40587]: pgmap v613: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 12193/128685 objects degraded (9.475%); 11455/128685 objects misplaced (8.902%); 44 MiB/s, 707 objects/s recovering 2024-03-26T21:42:25.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:25.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:25.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:25.533 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:25.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.672 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:25.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:25.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:25.808 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:25.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:25.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:25.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:26.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:26.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:26.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:26.341 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:26.443 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:26.471 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:26.573 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:26.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:26.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:26.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:26.862 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:26.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:26.989 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:26.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:27.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:27.120 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:27.180 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:27.180 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:27.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:27.256 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:27.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[40587]: pgmap v614: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 666 op/s; 11900/127806 objects degraded (9.311%); 11162/127806 objects misplaced (8.734%); 26 MiB/s, 415 objects/s recovering 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[40587]: osdmap e299: 8 total, 8 up, 7 in 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 11900/127806 objects degraded (9.311%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[29406]: pgmap v614: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 666 op/s; 11900/127806 objects degraded (9.311%); 11162/127806 objects misplaced (8.734%); 26 MiB/s, 415 objects/s recovering 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[29406]: osdmap e299: 8 total, 8 up, 7 in 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:27.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 11900/127806 objects degraded (9.311%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:27.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:27.393 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:27.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-mon[33346]: pgmap v614: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 3 active+undersized+degraded+remapped+backfill_toofull; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 666 op/s; 11900/127806 objects degraded (9.311%); 11162/127806 objects misplaced (8.734%); 26 MiB/s, 415 objects/s recovering 2024-03-26T21:42:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-mon[33346]: osdmap e299: 8 total, 8 up, 7 in 2024-03-26T21:42:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:27.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 11900/127806 objects degraded (9.311%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:27.496 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:27.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:27.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:27.668 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:27.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:27.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:27.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:27.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:42:27.942 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:27.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:28.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:28.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:28.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:28 smithi067 ceph-mon[33346]: osdmap e300: 8 total, 8 up, 7 in 2024-03-26T21:42:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:28.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:28.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:28.348 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:28.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:28.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:28 smithi055 ceph-mon[40587]: osdmap e300: 8 total, 8 up, 7 in 2024-03-26T21:42:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:28 smithi055 ceph-mon[29406]: osdmap e300: 8 total, 8 up, 7 in 2024-03-26T21:42:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:28.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:28.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:28.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:28.758 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:28.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:28.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:28.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:28.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:29.062 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:29.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:29.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:29.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:29 smithi055 ceph-mon[29406]: pgmap v617: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 10351/118875 objects degraded (8.707%); 10475/118875 objects misplaced (8.812%); 22 MiB/s, 345 objects/s recovering 2024-03-26T21:42:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:29 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:29 smithi055 ceph-mon[40587]: pgmap v617: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 10351/118875 objects degraded (8.707%); 10475/118875 objects misplaced (8.812%); 22 MiB/s, 345 objects/s recovering 2024-03-26T21:42:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:29 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:29.443 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:29.481 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:29 smithi067 ceph-mon[33346]: pgmap v617: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 2 active+undersized+degraded+remapped+backfill_toofull; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 10351/118875 objects degraded (8.707%); 10475/118875 objects misplaced (8.812%); 22 MiB/s, 345 objects/s recovering 2024-03-26T21:42:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:29 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:29.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:29.621 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:29.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:29.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:29.919 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:29.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:30.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:30.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:30.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.230 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:30.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:30.335 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:30.335 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:30.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:30.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:30.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:30.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:30.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:30.882 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:30.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:30.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:31.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:31.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:42:31.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:31.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:31 smithi067 ceph-mon[33346]: pgmap v618: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 31 MiB/s, 500 objects/s recovering 2024-03-26T21:42:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:31 smithi067 ceph-mon[33346]: osdmap e301: 8 total, 8 up, 7 in 2024-03-26T21:42:31.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.300 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:31.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:31.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:31.407 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:31.411 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:31 smithi055 ceph-mon[40587]: pgmap v618: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 31 MiB/s, 500 objects/s recovering 2024-03-26T21:42:31.412 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:31 smithi055 ceph-mon[40587]: osdmap e301: 8 total, 8 up, 7 in 2024-03-26T21:42:31.412 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:31 smithi055 ceph-mon[29406]: pgmap v618: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 31 MiB/s, 500 objects/s recovering 2024-03-26T21:42:31.412 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:31 smithi055 ceph-mon[29406]: osdmap e301: 8 total, 8 up, 7 in 2024-03-26T21:42:31.479 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:31.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:31.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:31.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:31.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:31.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:31.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:31.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:31.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:32.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:32.058 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:32.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:42:32.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.214 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:32.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[40587]: osdmap e302: 8 total, 8 up, 7 in 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 8614/113904 objects degraded (7.563%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[29406]: osdmap e302: 8 total, 8 up, 7 in 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 8614/113904 objects degraded (7.563%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:32.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:32.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:32.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:32 smithi067 ceph-mon[33346]: osdmap e302: 8 total, 8 up, 7 in 2024-03-26T21:42:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 8614/113904 objects degraded (7.563%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:32.489 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.532 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:32.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:32.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:32.668 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:32.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:32.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:32.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:32.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:33.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:33.136 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T21:42:33.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:33.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:33.367 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:33.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[40587]: pgmap v621: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 926 B/s rd, 0 B/s wr, 839 op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:42:33.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4293478370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[29406]: pgmap v621: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 926 B/s rd, 0 B/s wr, 839 op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:42:33.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4293478370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:33 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.418 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:33 smithi067 ceph-mon[33346]: pgmap v621: 148 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 139 active+clean, 1 active+undersized+degraded+remapped+backfill_toofull; 2.3 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 926 B/s rd, 0 B/s wr, 839 op/s; 8614/113904 objects degraded (7.563%); 9936/113904 objects misplaced (8.723%); 24 MiB/s, 381 objects/s recovering 2024-03-26T21:42:33.418 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4293478370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.418 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:33 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:42:33.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:33.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:33.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:33.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:33.768 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:33.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:33.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:34.009 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:34.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:34.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:42:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:34 smithi067 ceph-mon[33346]: osdmap e303: 8 total, 8 up, 7 in 2024-03-26T21:42:34.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:42:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:34 smithi055 ceph-mon[40587]: osdmap e303: 8 total, 8 up, 7 in 2024-03-26T21:42:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:42:34.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:34 smithi055 ceph-mon[29406]: osdmap e303: 8 total, 8 up, 7 in 2024-03-26T21:42:34.323 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:34.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:34.479 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T21:42:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:34.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:34.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:34.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:34.793 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:34.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:34.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:42:34.965 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:34.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:35.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:35.112 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:35.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:35 smithi055 ceph-mon[29406]: pgmap v623: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 140 active+clean; 2.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 735 op/s; 7472/105663 objects degraded (7.072%); 9384/105663 objects misplaced (8.881%); 20 MiB/s, 315 objects/s recovering 2024-03-26T21:42:35.183 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:35 smithi055 ceph-mon[40587]: pgmap v623: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 140 active+clean; 2.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 735 op/s; 7472/105663 objects degraded (7.072%); 9384/105663 objects misplaced (8.881%); 20 MiB/s, 315 objects/s recovering 2024-03-26T21:42:35.197 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:35.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:35.286 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:35.288 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.289 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:35.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:35.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:35.420 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:35 smithi067 ceph-mon[33346]: pgmap v623: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 140 active+clean; 2.1 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 735 op/s; 7472/105663 objects degraded (7.072%); 9384/105663 objects misplaced (8.881%); 20 MiB/s, 315 objects/s recovering 2024-03-26T21:42:35.429 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:35.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:35.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:35.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:35.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:35.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:35.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:35.987 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:35.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:36.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:36.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:36.169 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:42:36.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:36.262 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:36.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:36.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:42:36.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:36.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:42:36.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:42:36.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:36.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:36.821 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:36.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:36.954 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:36.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:37.055 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:37.081 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:37.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:37.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:37.385 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: pgmap v624: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 5912/101562 objects degraded (5.821%); 8359/101562 objects misplaced (8.230%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:42:37.385 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:37.385 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:37.385 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:42:37.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: osdmap e304: 8 total, 8 up, 7 in 2024-03-26T21:42:37.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:42:37.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:42:37.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5912/101562 objects degraded (5.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:37.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-mon[33346]: osdmap e305: 8 total, 8 up, 7 in 2024-03-26T21:42:37.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:37.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:37.564 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:37.564 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: pgmap v624: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 5912/101562 objects degraded (5.821%); 8359/101562 objects misplaced (8.230%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: osdmap e304: 8 total, 8 up, 7 in 2024-03-26T21:42:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5912/101562 objects degraded (5.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[40587]: osdmap e305: 8 total, 8 up, 7 in 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: pgmap v624: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 5912/101562 objects degraded (5.821%); 8359/101562 objects misplaced (8.230%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/875724968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: osdmap e304: 8 total, 8 up, 7 in 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5912/101562 objects degraded (5.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:37 smithi055 ceph-mon[29406]: osdmap e305: 8 total, 8 up, 7 in 2024-03-26T21:42:37.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:37.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:37.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:37.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:37.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:37.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:38.019 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:38.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:38.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:38.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:38.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:38 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:38 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:38 smithi067 ceph-mon[33346]: osdmap e306: 8 total, 8 up, 7 in 2024-03-26T21:42:38.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:38.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[40587]: osdmap e306: 8 total, 8 up, 7 in 2024-03-26T21:42:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-mon[29406]: osdmap e306: 8 total, 8 up, 7 in 2024-03-26T21:42:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:38.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:38.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:38.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:38.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:38.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:38.987 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:38.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:39.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:39.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:39.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:39.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:39 smithi055 ceph-mon[40587]: pgmap v627: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 5545/96420 objects degraded (5.751%); 7992/96420 objects misplaced (8.289%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:42:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:39 smithi055 ceph-mon[29406]: pgmap v627: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 5545/96420 objects degraded (5.751%); 7992/96420 objects misplaced (8.289%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:42:39.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:39.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:39.409 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:39 smithi067 ceph-mon[33346]: pgmap v627: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 971 op/s; 5545/96420 objects degraded (5.751%); 7992/96420 objects misplaced (8.289%); 20 MiB/s, 318 objects/s recovering 2024-03-26T21:42:39.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:39.548 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:39.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:39.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:39.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:39.813 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:39.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:39.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:39.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:40.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:40.075 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:40.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:40.203 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:40.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:40.328 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:40.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:40.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:40.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:40.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.753 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:40.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:40.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:40.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:40.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:41.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:41.171 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:41.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:41.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:41.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:41.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:41 smithi067 ceph-mon[33346]: pgmap v629: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 28 MiB/s, 443 objects/s recovering 2024-03-26T21:42:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:41 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:41 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:41 smithi067 ceph-mon[33346]: osdmap e307: 8 total, 8 up, 7 in 2024-03-26T21:42:41.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[40587]: pgmap v629: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 28 MiB/s, 443 objects/s recovering 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[40587]: osdmap e307: 8 total, 8 up, 7 in 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[29406]: pgmap v629: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 28 MiB/s, 443 objects/s recovering 2024-03-26T21:42:41.554 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:41.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:41.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:41 smithi055 ceph-mon[29406]: osdmap e307: 8 total, 8 up, 7 in 2024-03-26T21:42:41.577 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:41.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:41.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:41.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:41.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:41.986 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:41.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:42.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:42.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:42.259 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:42.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:42.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 3718/86835 objects degraded (4.282%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:42.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[29406]: osdmap e308: 8 total, 8 up, 7 in 2024-03-26T21:42:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 3718/86835 objects degraded (4.282%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:42 smithi055 ceph-mon[40587]: osdmap e308: 8 total, 8 up, 7 in 2024-03-26T21:42:42.361 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:42.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:42.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:42 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:42:42.538 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 3718/86835 objects degraded (4.282%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:42.538 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:42 smithi067 ceph-mon[33346]: osdmap e308: 8 total, 8 up, 7 in 2024-03-26T21:42:42.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:42.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:42.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:42:42.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:42.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:42.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:42.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:42.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:43.032 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:43.088 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:42:43.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:43.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:42:43.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:43 smithi055 ceph-mon[40587]: pgmap v631: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 907 op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 16 MiB/s, 261 objects/s recovering 2024-03-26T21:42:43.266 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:43 smithi055 ceph-mon[29406]: pgmap v631: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 907 op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 16 MiB/s, 261 objects/s recovering 2024-03-26T21:42:43.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:43.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:43.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:43 smithi067 ceph-mon[33346]: pgmap v631: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 907 op/s; 3718/86835 objects degraded (4.282%); 6603/86835 objects misplaced (7.604%); 16 MiB/s, 261 objects/s recovering 2024-03-26T21:42:43.503 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:42:43.503 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:42:43.503 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:42:43.503 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:42:43.532 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:42:43.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:43.570 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:42:43.584 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:43.599 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:42:43.633 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:42:43.635 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:42:43.636 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:42:43.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:43.709 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:43.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:43.910 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:43.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:43.971 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:43.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:44.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:44.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:44.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:44.249 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T21:42:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:42:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:42:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: osdmap e309: 8 total, 8 up, 7 in 2024-03-26T21:42:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:44.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4089551016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3066346367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: osdmap e309: 8 total, 8 up, 7 in 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4089551016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3066346367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:44 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:44.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:44.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: osdmap e309: 8 total, 8 up, 7 in 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4089551016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3066346367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:44 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:42:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.503 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:44.604 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:44.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:44.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:44.781 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:44.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:44.909 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:44.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:45.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:45.045 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:45.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:45.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:45.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:45.321 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:45.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:45.462 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[29406]: pgmap v634: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.6 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 788 op/s; 2313/78282 objects degraded (2.955%); 5435/78282 objects misplaced (6.943%); 21 MiB/s, 328 objects/s recovering 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[29406]: osdmap e310: 8 total, 8 up, 7 in 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[40587]: pgmap v634: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.6 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 788 op/s; 2313/78282 objects degraded (2.955%); 5435/78282 objects misplaced (6.943%); 21 MiB/s, 328 objects/s recovering 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[40587]: osdmap e310: 8 total, 8 up, 7 in 2024-03-26T21:42:45.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:42:45.514 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:45 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:42:45.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:45.601 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:45.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.739 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:45.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:45 smithi067 ceph-mon[33346]: pgmap v634: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.6 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 788 op/s; 2313/78282 objects degraded (2.955%); 5435/78282 objects misplaced (6.943%); 21 MiB/s, 328 objects/s recovering 2024-03-26T21:42:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:42:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:45 smithi067 ceph-mon[33346]: osdmap e310: 8 total, 8 up, 7 in 2024-03-26T21:42:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:45 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:42:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:45 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:42:45.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:45.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:45.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T21:42:45.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:46.018 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:46.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:46.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:46.273 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:46.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:46.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:46.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:46.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:46 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:42:46.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:46 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:42:46.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:42:46.528 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:46 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:42:46.529 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:46 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:42:46.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:46 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:42:46.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:46 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:42:46.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:46.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:46.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:46.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:46.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:46.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:46.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:47.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:47.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:47.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:47.221 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:47.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:47.354 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:47.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:47.387 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-mon[33346]: pgmap v636: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 950 B/s rd, 0 B/s wr, 793 op/s; 1849/74022 objects degraded (2.498%); 4721/74022 objects misplaced (6.378%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:42:47.387 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-mon[33346]: osdmap e311: 8 total, 8 up, 7 in 2024-03-26T21:42:47.387 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1849/74022 objects degraded (2.498%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:47.387 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-mon[33346]: osdmap e312: 8 total, 8 up, 7 in 2024-03-26T21:42:47.387 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:47.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166762 9.d 9.8ef975ad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.615189+0000", 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.947021480000004, 2024-03-26T21:42:47.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99033887600000003, 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166762 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615189+0000", 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615189+0000", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615190+0000", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2270000000000001e-06 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615202+0000", 2024-03-26T21:42:47.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1276e-05 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615219+0000", 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7201999999999999e-05 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615230+0000", 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0855e-05 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615306+0000", 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6215000000000001e-05 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.615318+0000", 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1945e-05 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605514+0000", 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99019644799999995 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605528+0000", 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3708e-05 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166779 9.e 9.4a4fcfbe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.620848+0000", 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.941363042000006, 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97858629799999997, 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166779 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620848+0000", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620848+0000", 2024-03-26T21:42:47.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620849+0000", 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.638e-06 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620856+0000", 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4200000000000004e-06 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620868+0000", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2109e-05 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620876+0000", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620946+0000", 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9568999999999997e-05 2024-03-26T21:42:47.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.620958+0000", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.203e-05 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.599410+0000", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97845225999999996 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.599434+0000", 2024-03-26T21:42:47.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4012999999999999e-05 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166781 9.d 9.1c199bbd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.624659+0000", 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.937551232000004, 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98091037400000003, 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166781 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624659+0000", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624659+0000", 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624661+0000", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2610000000000001e-06 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624665+0000", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5539999999999999e-06 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624675+0000", 2024-03-26T21:42:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8980000000000008e-06 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624682+0000", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3950000000000003e-06 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624733+0000", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0710000000000001e-05 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624741+0000", 2024-03-26T21:42:47.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8120000000000008e-06 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605558+0000", 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98081667299999997 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.573 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[29406]: pgmap v636: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 950 B/s rd, 0 B/s wr, 793 op/s; 1849/74022 objects degraded (2.498%); 4721/74022 objects misplaced (6.378%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:42:47.573 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[29406]: osdmap e311: 8 total, 8 up, 7 in 2024-03-26T21:42:47.573 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1849/74022 objects degraded (2.498%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:47.573 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[29406]: osdmap e312: 8 total, 8 up, 7 in 2024-03-26T21:42:47.573 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:47.574 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[40587]: pgmap v636: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.5 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 950 B/s rd, 0 B/s wr, 793 op/s; 1849/74022 objects degraded (2.498%); 4721/74022 objects misplaced (6.378%); 17 MiB/s, 269 objects/s recovering 2024-03-26T21:42:47.574 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[40587]: osdmap e311: 8 total, 8 up, 7 in 2024-03-26T21:42:47.574 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1849/74022 objects degraded (2.498%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T21:42:47.574 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[40587]: osdmap e312: 8 total, 8 up, 7 in 2024-03-26T21:42:47.574 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:42:47.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.605570+0000", 2024-03-26T21:42:47.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2071e-05 2024-03-26T21:42:47.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.574 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2916 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94208173336832 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.593744+0000", 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.968466942999996, 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0315166739999999, 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:42:47.575 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2916 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593744+0000", 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593744+0000", 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593745+0000", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2720000000000001e-06 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593749+0000", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0260000000000001e-06 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593758+0000", 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2890000000000003e-06 2024-03-26T21:42:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593767+0000", 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7849999999999997e-06 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593909+0000", 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141941 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.593921+0000", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1722999999999999e-05 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.613163+0000", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0192425869999999 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.613344+0000", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181218 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.625260+0000", 2024-03-26T21:42:47.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011915833000000001 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214505 9.d 9:b46ec496:::benchmark_data_smithi055_135863_object83854:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.772317+0000", 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.789893295999999, 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.047058437, 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214505 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772317+0000", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772317+0000", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772319+0000", 2024-03-26T21:42:47.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9010000000000001e-06 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772326+0000", 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4590000000000003e-06 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772338+0000", 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500999999999999e-05 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772347+0000", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774130+0000", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001783651 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774138+0000", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3239999999999997e-06 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780456+0000", 2024-03-26T21:42:47.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063179810000000003 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780465+0000", 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8079999999999999e-06 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819314+0000", 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0388484979999999 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819376+0000", 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1834999999999993e-05 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214507 9.d 9:bca6c72a:::benchmark_data_smithi055_135863_object83856:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.772371+0000", 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.789839539999999, 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0473891559999999, 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214507 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772371+0000", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772371+0000", 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772372+0000", 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9100000000000004e-07 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772383+0000", 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1223000000000001e-05 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772390+0000", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8870000000000003e-06 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772395+0000", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5860000000000004e-06 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774185+0000", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017895299999999999 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774190+0000", 2024-03-26T21:42:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2789999999999998e-06 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780490+0000", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063000699999999996 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780498+0000", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7640000000000005e-06 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819689+0000", 2024-03-26T21:42:47.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.039191526 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819760+0000", 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-05 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214511 9.d 9:b66b12c0:::benchmark_data_smithi055_135863_object83860:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.772403+0000", 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.789807559, 2024-03-26T21:42:47.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.047466429, 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214511 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772403+0000", 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772403+0000", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772403+0000", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7e-07 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772406+0000", 2024-03-26T21:42:47.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2510000000000001e-06 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772410+0000", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6319999999999999e-06 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772415+0000", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9180000000000002e-06 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774296+0000", 2024-03-26T21:42:47.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018811699999999999 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774303+0000", 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6150000000000002e-06 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780523+0000", 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062204310000000002 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780530+0000", 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6080000000000001e-06 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819804+0000", 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.039274464 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819869+0000", 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4969999999999996e-05 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214513 9.d 9:b3b662f8:::benchmark_data_smithi055_135863_object83862:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.772421+0000", 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.789789894999998, 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.047550947, 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:42:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214513 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772421+0000", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772421+0000", 2024-03-26T21:42:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772421+0000", 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1100000000000002e-07 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772423+0000", 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4600000000000002e-06 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772428+0000", 2024-03-26T21:42:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5229999999999997e-06 2024-03-26T21:42:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.772433+0000", 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5430000000000004e-06 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774791+0000", 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023586589999999999 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.774798+0000", 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3260000000000001e-06 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780802+0000", 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060043249999999996 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.780809+0000", 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4170000000000003e-06 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819906+0000", 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0390972510000001 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.819972+0000", 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5131999999999999e-05 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214526 9.d 9:bc7c5466:::benchmark_data_smithi055_135863_object83875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.786906+0000", 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.775304520999999, 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0394242899999999, 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214526 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786906+0000", 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786906+0000", 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786908+0000", 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.697e-06 2024-03-26T21:42:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786916+0000", 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5269999999999998e-06 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786932+0000", 2024-03-26T21:42:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6144999999999999e-05 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.786942+0000", 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6749999999999997e-06 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.787007+0000", 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4993999999999995e-05 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.787019+0000", 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1775e-05 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.820007+0000", 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0329880359999999 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.820109+0000", 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102181 2024-03-26T21:42:47.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.820201+0000", 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2275000000000007e-05 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826154+0000", 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059528899999999997 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826224+0000", 2024-03-26T21:42:47.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9676000000000002e-05 2024-03-26T21:42:47.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826242+0000", 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.836e-05 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826293+0000", 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0967000000000002e-05 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826330+0000", 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7092000000000003e-05 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214529 9.d 9:bf4ed282:::benchmark_data_smithi055_135863_object83878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.791656+0000", 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.770554270000002, 2024-03-26T21:42:47.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.034859516, 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214529 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.617 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791656+0000", 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791656+0000", 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791658+0000", 2024-03-26T21:42:47.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4839999999999999e-06 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791664+0000", 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3169999999999998e-06 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791682+0000", 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7507e-05 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791691+0000", 2024-03-26T21:42:47.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0280000000000006e-06 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791779+0000", 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8706000000000001e-05 2024-03-26T21:42:47.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.791801+0000", 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2062999999999999e-05 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.822217+0000", 2024-03-26T21:42:47.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0304157899999999 2024-03-26T21:42:47.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.822333+0000", 2024-03-26T21:42:47.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011601500000000001 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.822440+0000", 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106542 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826377+0000", 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039371040000000003 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826428+0000", 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1103999999999997e-05 2024-03-26T21:42:47.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826460+0000", 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.188e-05 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826491+0000", 2024-03-26T21:42:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0766000000000003e-05 2024-03-26T21:42:47.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826516+0000", 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5210000000000001e-05 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.635 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214538 9.d 9:bd55f48b:::benchmark_data_smithi055_135863_object83887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.636 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.805247+0000", 2024-03-26T21:42:47.636 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.756963314, 2024-03-26T21:42:47.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.022809297, 2024-03-26T21:42:47.636 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.637 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.637 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.637 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.638 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.638 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214538 2024-03-26T21:42:47.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.638 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805247+0000", 2024-03-26T21:42:47.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805247+0000", 2024-03-26T21:42:47.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805249+0000", 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5009999999999999e-06 2024-03-26T21:42:47.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805255+0000", 2024-03-26T21:42:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0020000000000003e-06 2024-03-26T21:42:47.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805267+0000", 2024-03-26T21:42:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2643000000000001e-05 2024-03-26T21:42:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805275+0000", 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3950000000000003e-06 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805691+0000", 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041631600000000002 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.805701+0000", 2024-03-26T21:42:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6690000000000001e-06 2024-03-26T21:42:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.824249+0000", 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018548067 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.824360+0000", 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011147200000000001 2024-03-26T21:42:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.824467+0000", 2024-03-26T21:42:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106548 2024-03-26T21:42:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826420+0000", 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019532009999999999 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826432+0000", 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1958e-05 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.827990+0000", 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558378 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.828022+0000", 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1507999999999997e-05 2024-03-26T21:42:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.828057+0000", 2024-03-26T21:42:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4638999999999998e-05 2024-03-26T21:42:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214540 9.d 9:bbcb20c3:::benchmark_data_smithi055_135863_object83889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.806104+0000", 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.756106636999998, 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.026129106, 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214540 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806104+0000", 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806104+0000", 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806105+0000", 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-07 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806108+0000", 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6770000000000001e-06 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806118+0000", 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0210999999999999e-05 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806126+0000", 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5190000000000003e-06 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806162+0000", 2024-03-26T21:42:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5565000000000003e-05 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.806167+0000", 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.955e-06 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826548+0000", 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0203817980000001 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826666+0000", 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011784499999999999 2024-03-26T21:42:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.826787+0000", 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120815 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832041+0000", 2024-03-26T21:42:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052538359999999996 2024-03-26T21:42:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832124+0000", 2024-03-26T21:42:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3344999999999996e-05 2024-03-26T21:42:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832138+0000", 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3423e-05 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832190+0000", 2024-03-26T21:42:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2058999999999997e-05 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832233+0000", 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3409999999999999e-05 2024-03-26T21:42:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214557 9.d 9:b5c3e339:::benchmark_data_smithi055_135863_object83906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.827353+0000", 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.734858036999999, 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.010244967, 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214557 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.827353+0000", 2024-03-26T21:42:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.827353+0000", 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.827356+0000", 2024-03-26T21:42:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.14e-06 2024-03-26T21:42:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.827369+0000", 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3626e-05 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.862176+0000", 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034806549999999999 2024-03-26T21:42:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.862190+0000", 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3805000000000001e-05 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.862887+0000", 2024-03-26T21:42:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069691100000000004 2024-03-26T21:42:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.862896+0000", 2024-03-26T21:42:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7110000000000007e-06 2024-03-26T21:42:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.828097+0000", 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96520101999999997 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.828176+0000", 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8734999999999997e-05 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.828253+0000", 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7371000000000002e-05 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.834867+0000", 2024-03-26T21:42:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066135339999999999 2024-03-26T21:42:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.834904+0000", 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6721000000000003e-05 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837486+0000", 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025827960000000001 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837544+0000", 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7907e-05 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837598+0000", 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3140000000000003e-05 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214581 9.d 9:b2289358:::benchmark_data_smithi055_135863_object83930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.883547+0000", 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.678663355000001, 2024-03-26T21:42:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95423418900000001, 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214581 2024-03-26T21:42:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883547+0000", 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883547+0000", 2024-03-26T21:42:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883549+0000", 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4100000000000001e-06 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883556+0000", 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8800000000000002e-06 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883566+0000", 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0548e-05 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883574+0000", 2024-03-26T21:42:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2819999999999996e-06 2024-03-26T21:42:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883625+0000", 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1044000000000001e-05 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.883637+0000", 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1182999999999999e-05 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.830158+0000", 2024-03-26T21:42:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94652117800000002 2024-03-26T21:42:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.830281+0000", 2024-03-26T21:42:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123359 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.830408+0000", 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012661500000000001 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.834915+0000", 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045072369999999999 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837650+0000", 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027351939999999998 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837691+0000", 2024-03-26T21:42:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0794000000000003e-05 2024-03-26T21:42:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837743+0000", 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1799000000000003e-05 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837781+0000", 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8665999999999999e-05 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214582 9.d 9:b2b72bed:::benchmark_data_smithi055_135863_object83931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.884050+0000", 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.678161039999999, 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96081052099999997, 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214582 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884050+0000", 2024-03-26T21:42:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884050+0000", 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884051+0000", 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9600000000000008e-07 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884059+0000", 2024-03-26T21:42:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9869999999999999e-06 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884070+0000", 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.188e-05 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884078+0000", 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.678e-06 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884145+0000", 2024-03-26T21:42:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6490999999999997e-05 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.884153+0000", 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9910000000000002e-06 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832274+0000", 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94812149800000001 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832386+0000", 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111556 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.832506+0000", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120043 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.839994+0000", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074879270000000001 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.840005+0000", 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1780000000000001e-05 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844754+0000", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047483919999999997 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844812+0000", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8114e-05 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844860+0000", 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8188e-05 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214599 9.d 9:b4dec8e9:::benchmark_data_smithi055_135863_object83948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.901788+0000", 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.660422840999999, 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94320424599999997, 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214599 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901788+0000", 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901788+0000", 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901790+0000", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.99e-06 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901796+0000", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5860000000000001e-06 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901810+0000", 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4154e-05 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901818+0000", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8509999999999999e-06 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901884+0000", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5535000000000002e-05 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.901894+0000", 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0504e-05 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.834937+0000", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.933042496 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.835482+0000", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054491999999999995 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.835588+0000", 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010632399999999999 2024-03-26T21:42:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.840054+0000", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044661509999999998 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844664+0000", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046097999999999998 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844901+0000", 2024-03-26T21:42:47.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023733599999999999 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844955+0000", 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3282999999999998e-05 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844992+0000", 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7316e-05 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214628 9.d 9:b1a35bf3:::benchmark_data_smithi055_135863_object83977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.942727+0000", 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.619484031999999, 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.902484854, 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214628 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942727+0000", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942727+0000", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942728+0000", 2024-03-26T21:42:47.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5489999999999999e-06 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942734+0000", 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2639999999999997e-06 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942746+0000", 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.133e-05 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.942753+0000", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4590000000000001e-06 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.943583+0000", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083029000000000002 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.943593+0000", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7170000000000003e-06 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837810+0000", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.89421650799999997 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.837936+0000", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126725 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.838116+0000", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017996 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.844713+0000", 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065969590000000003 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845050+0000", 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033632600000000002 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845072+0000", 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2001000000000002e-05 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845168+0000", 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6096999999999998e-05 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845211+0000", 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3668999999999998e-05 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214629 9.d 9:b4dc0875:::benchmark_data_smithi055_135863_object83978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.723 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.944364+0000", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.617846155999999, 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.91678008, 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214629 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944364+0000", 2024-03-26T21:42:47.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944364+0000", 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944365+0000", 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2699999999999998e-07 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944370+0000", 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2930000000000002e-06 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944378+0000", 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5990000000000002e-06 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944385+0000", 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3810000000000004e-06 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944428+0000", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3062999999999997e-05 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.944437+0000", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.183e-06 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.840080+0000", 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.89564338600000004 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.840227+0000", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146391 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.840326+0000", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9677000000000001e-05 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.849863+0000", 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095363989999999992 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.854640+0000", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047770149999999999 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861038+0000", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063986169999999997 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861097+0000", 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8996999999999998e-05 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861145+0000", 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7151999999999998e-05 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214636 9.d 9:b20fa121:::benchmark_data_smithi055_135863_object83985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.956996+0000", 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.605215053999999, 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.90427603599999995, 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.730 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214636 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.956996+0000", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.956996+0000", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.956997+0000", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.282e-06 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957001+0000", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.437e-06 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957011+0000", 2024-03-26T21:42:47.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0012e-05 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957017+0000", 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0270000000000001e-06 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957079+0000", 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1789000000000001e-05 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957090+0000", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1021e-05 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.842329+0000", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.88523852000000003 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.842435+0000", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106186 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.842536+0000", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100963 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.854679+0000", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012143584000000001 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.856098+0000", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014184130000000001 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861188+0000", 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050901050000000002 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861235+0000", 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7537000000000002e-05 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.861272+0000", 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6159999999999999e-05 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:214637 9.d 9:b92dbce0:::benchmark_data_smithi055_135863_object83986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:16.957566+0000", 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.604644238999999, 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.90756428200000006, 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 214637 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957566+0000", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957566+0000", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957567+0000", 2024-03-26T21:42:47.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5499999999999998e-07 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957570+0000", 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4369999999999999e-06 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957579+0000", 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5830000000000007e-06 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:42:47.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957585+0000", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7250000000000002e-06 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957622+0000", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7725000000000002e-05 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:16.957628+0000", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5099999999999998e-06 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:42:47.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845284+0000", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.88765559400000005 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845362+0000", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8726999999999993e-05 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.845503+0000", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014066099999999999 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.854712+0000", 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092089879999999995 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.862224+0000", 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075116080000000003 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.864938+0000", 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027145450000000001 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.865076+0000", 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013829700000000001 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:17.865131+0000", 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4227000000000001e-05 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:42:47.743 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:42:47.772 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:47.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:47.916 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:47.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:48.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:48.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:48.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:48.327 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:48.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:48 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:48 smithi067 ceph-mon[33346]: osdmap e313: 8 total, 8 up, 7 in 2024-03-26T21:42:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:48 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:48 smithi055 ceph-mon[29406]: osdmap e313: 8 total, 8 up, 7 in 2024-03-26T21:42:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:48 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:48 smithi055 ceph-mon[40587]: osdmap e313: 8 total, 8 up, 7 in 2024-03-26T21:42:48.567 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:48.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:48.651 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:42:48.702 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:48.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:48.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:48.875 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:48.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:48.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:49.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:49.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:49.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:49.281 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:49.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:49.395 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:49.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:49.521 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:49.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:49 smithi055 ceph-mon[40587]: pgmap v639: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 1098/70392 objects degraded (1.560%); 3970/70392 objects misplaced (5.640%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:42:49.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:49 smithi055 ceph-mon[40587]: osdmap e314: 8 total, 8 up, 7 in 2024-03-26T21:42:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:49 smithi055 ceph-mon[29406]: pgmap v639: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 1098/70392 objects degraded (1.560%); 3970/70392 objects misplaced (5.640%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:42:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:49 smithi055 ceph-mon[29406]: osdmap e314: 8 total, 8 up, 7 in 2024-03-26T21:42:49.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:49.650 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:49 smithi067 ceph-mon[33346]: pgmap v639: 148 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 1098/70392 objects degraded (1.560%); 3970/70392 objects misplaced (5.640%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:42:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:49 smithi067 ceph-mon[33346]: osdmap e314: 8 total, 8 up, 7 in 2024-03-26T21:42:49.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:49.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:49.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:49.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:50.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:50.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:50.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:50.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:50.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:50.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:50.436 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:50.538 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:42:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[40587]: osdmap e315: 8 total, 8 up, 7 in 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:42:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:50 smithi055 ceph-mon[29406]: osdmap e315: 8 total, 8 up, 7 in 2024-03-26T21:42:50.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:50.611 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:50.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:42:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:50 smithi067 ceph-mon[33346]: osdmap e315: 8 total, 8 up, 7 in 2024-03-26T21:42:50.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:50.855 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:50.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:50.970 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:50.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:51.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:51.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:51.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:51.324 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:51.348 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:51.352 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:42:51.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:51.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:51.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: pgmap v642: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 40 MiB/s, 632 objects/s recovering 2024-03-26T21:42:51.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:42:51.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: 9.c scrub starts 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: 9.c scrub ok 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[40587]: osdmap e316: 8 total, 8 up, 7 in 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: pgmap v642: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 40 MiB/s, 632 objects/s recovering 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: 9.c scrub starts 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: 9.c scrub ok 2024-03-26T21:42:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:51 smithi055 ceph-mon[29406]: osdmap e316: 8 total, 8 up, 7 in 2024-03-26T21:42:51.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:51.598 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:51.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:51.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: pgmap v642: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 857 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 40 MiB/s, 632 objects/s recovering 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: 9.c scrub starts 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: 9.c scrub ok 2024-03-26T21:42:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:51 smithi067 ceph-mon[33346]: osdmap e316: 8 total, 8 up, 7 in 2024-03-26T21:42:51.801 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:42:51.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.857 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:51.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:51.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:51.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:51.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:52.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:52.134 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:52.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:52.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:52.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:52.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:52.551 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:52.653 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:52.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/564122188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:42:52.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:42:52.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:52.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:52 smithi067 ceph-mon[33346]: osdmap e317: 8 total, 8 up, 7 in 2024-03-26T21:42:52.768 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:42:52.768 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:42:52.768 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/564122188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:52.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[29406]: osdmap e317: 8 total, 8 up, 7 in 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/564122188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:42:52.771 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:42:52.771 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:42:52.771 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:42:52.771 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:52 smithi055 ceph-mon[40587]: osdmap e317: 8 total, 8 up, 7 in 2024-03-26T21:42:52.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:52.821 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:52.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:52.955 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:52.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:53.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:53.092 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:53.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:53.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:53.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:53.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:53.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:53.502 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:53.590 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: pgmap v645: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 555 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 30 MiB/s, 479 objects/s recovering 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: 9.1 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: pgmap v645: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 555 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 30 MiB/s, 479 objects/s recovering 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: 9.1 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:42:53.591 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:53 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:42:53.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: pgmap v645: 148 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 555 op/s; 343/63741 objects degraded (0.538%); 1675/63741 objects misplaced (2.628%); 30 MiB/s, 479 objects/s recovering 2024-03-26T21:42:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:42:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: 9.1 scrub starts 2024-03-26T21:42:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:42:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:42:53.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:53 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:42:53.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:53.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:53.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:53.916 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:53.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:54.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:54.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:54.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:54.185 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:54.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:54.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.458 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:54.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:54.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:54.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:54.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:54.734 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: 9.1 scrub ok 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: 9.a scrub starts 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: 9.a scrub ok 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:54 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:54.811 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:42:54.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: 9.1 scrub ok 2024-03-26T21:42:54.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:42:54.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: 9.a scrub starts 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: 9.a scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: 9.1 scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: 9.a scrub starts 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: 9.a scrub ok 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:54.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:54 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:54.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:54.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:54.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:54.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:55.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:55.111 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.163 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:55.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:55.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: pgmap v647: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 410 op/s; 456/57852 objects misplaced (0.788%); 15 MiB/s, 242 objects/s recovering 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 5.8 deep-scrub starts 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 5.8 deep-scrub ok 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 4.0 deep-scrub starts 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 4.0 deep-scrub ok 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: osdmap e318: 8 total, 8 up, 7 in 2024-03-26T21:42:55.437 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: pgmap v647: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 410 op/s; 456/57852 objects misplaced (0.788%); 15 MiB/s, 242 objects/s recovering 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 5.8 deep-scrub starts 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 5.8 deep-scrub ok 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 4.0 deep-scrub starts 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 4.0 deep-scrub ok 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: osdmap e318: 8 total, 8 up, 7 in 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:42:55.438 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:55 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:42:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:55.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:55.575 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:55.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.717 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: pgmap v647: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 410 op/s; 456/57852 objects misplaced (0.788%); 15 MiB/s, 242 objects/s recovering 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 5.8 deep-scrub starts 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 5.8 deep-scrub ok 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 343/63741 objects degraded (0.538%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 4.0 deep-scrub starts 2024-03-26T21:42:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 4.0 deep-scrub ok 2024-03-26T21:42:55.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: osdmap e318: 8 total, 8 up, 7 in 2024-03-26T21:42:55.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:42:55.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:55 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:42:55.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:55.854 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:55.956 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:55.993 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:55.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:56.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:56.134 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:56.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:42:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.268 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:56.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:42:56.365 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T21:42:56.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:56.502 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:56.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:56.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:56.676 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:56.728 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:42:56.728 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: 5.a deep-scrub starts 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: 5.a deep-scrub ok 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/71543293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: 5.a deep-scrub starts 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: 5.a deep-scrub ok 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/71543293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.729 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:56 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: 5.a deep-scrub starts 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: 5.a deep-scrub ok 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/71543293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:56 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:42:56.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:56.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:56.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:56.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:56.958 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:42:56.993 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:42:57.026 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:42:57.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:57.063 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:42:57.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:57.100 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:42:57.131 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:42:57.165 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:42:57.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:57.204 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:42:57.207 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:42:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:57.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.372 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:57.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:57.473 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:57.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:57.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:57.621 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: pgmap v649: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 524 op/s; 17 MiB/s, 276 objects/s recovering 2024-03-26T21:42:57.621 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 9.7 scrub starts 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 3.1b deep-scrub starts 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 3.1b deep-scrub ok 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: osdmap e319: 8 total, 8 up, 7 in 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: 9.7 scrub ok 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: osdmap e320: 8 total, 8 up, 7 in 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3866538905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: pgmap v649: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 524 op/s; 17 MiB/s, 276 objects/s recovering 2024-03-26T21:42:57.622 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 9.7 scrub starts 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 3.1b deep-scrub starts 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 3.1b deep-scrub ok 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: osdmap e319: 8 total, 8 up, 7 in 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: 9.7 scrub ok 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: osdmap e320: 8 total, 8 up, 7 in 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3866538905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:57.623 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:57 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:57.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:57.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:42:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: pgmap v649: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 524 op/s; 17 MiB/s, 276 objects/s recovering 2024-03-26T21:42:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 9.7 scrub starts 2024-03-26T21:42:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 3.1b deep-scrub starts 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 3.1b deep-scrub ok 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: osdmap e319: 8 total, 8 up, 7 in 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: 9.7 scrub ok 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: osdmap e320: 8 total, 8 up, 7 in 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3866538905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:42:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:57 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:42:57.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:57.784 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:57.886 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:57.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:57.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:58.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:58.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:58.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:42:58.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:42:58.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:58.463 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T21:42:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:58.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:42:58.607 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:42:58.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:42:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:42:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: pgmap v652: 148 pgs: 148 active+clean; 1000 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 851 op/s; 14 MiB/s, 218 objects/s recovering 2024-03-26T21:42:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:42:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:42:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:58 smithi067 ceph-mon[33346]: osdmap e321: 8 total, 8 up, 7 in 2024-03-26T21:42:58.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: pgmap v652: 148 pgs: 148 active+clean; 1000 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 851 op/s; 14 MiB/s, 218 objects/s recovering 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[40587]: osdmap e321: 8 total, 8 up, 7 in 2024-03-26T21:42:58.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: pgmap v652: 148 pgs: 148 active+clean; 1000 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 851 op/s; 14 MiB/s, 218 objects/s recovering 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:42:58.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-mon[29406]: osdmap e321: 8 total, 8 up, 7 in 2024-03-26T21:42:58.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:58.883 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:58.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:58.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:59.040 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:59.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:59.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:59.211 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:42:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:42:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:42:59.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:42:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:59.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:42:59.409 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:42:59.409 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:42:59.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:42:59.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:42:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:42:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:42:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:42:59 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:42:59.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:42:59.608 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:42:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:42:59.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:59 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:42:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:59 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:42:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:59 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:42:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:42:59 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:42:59.749 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:59.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:42:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:42:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:42:59.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:42:59.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:00.022 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:00.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.159 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:00.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:00.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:00.286 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:00.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:00.317 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:00.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:00.453 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:00.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:00.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:43:00.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:43:00.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:43:00.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[40587]: pgmap v654: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 976 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:43:00.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:00 smithi055 ceph-mon[29406]: pgmap v654: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 976 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T21:43:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:00.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:00.729 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:43:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:00 smithi067 ceph-mon[33346]: pgmap v654: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 976 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T21:43:00.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:00.869 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:00.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:00.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:43:00.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:01.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:01.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:01.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:01.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:01.390 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:01.433 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:01.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:43:01.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:01.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:01 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:43:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:01 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:43:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:01 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:43:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:01 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:43:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:43:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:01.863 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:01.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:01.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:01.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:02.033 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:02.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:02.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:02.299 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:43:02.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:02.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:02.556 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:02.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:02.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:02.716 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: 5.b deep-scrub starts 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: 5.b deep-scrub ok 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: pgmap v655: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: osdmap e322: 8 total, 8 up, 7 in 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:02.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:43:02.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: 5.b deep-scrub starts 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: 5.b deep-scrub ok 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: pgmap v655: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: osdmap e322: 8 total, 8 up, 7 in 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: 5.b deep-scrub starts 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: 5.b deep-scrub ok 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: pgmap v655: 148 pgs: 148 active+clean; 721 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T21:43:02.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: osdmap e322: 8 total, 8 up, 7 in 2024-03-26T21:43:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:02.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:02.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:02.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:02.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:02.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:03.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:03.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:03.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:03.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:03.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:03.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:03.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.541 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:03.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:03.551 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:03 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:43:03.551 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:03 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:43:03.551 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:03 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:43:03.551 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:03 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:43:03.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:03.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:43:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:43:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:43:03.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:43:03.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:43:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:43:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:43:03.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:03 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:43:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:03.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:03.978 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:03.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:04.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:04.114 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:04.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:04.244 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:04.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:04.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:04 smithi067 ceph-mon[33346]: pgmap v657: 148 pgs: 148 active+clean; 448 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-26T21:43:04.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:04.524 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:04.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:04.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:04.740 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:04 smithi055 ceph-mon[29406]: pgmap v657: 148 pgs: 148 active+clean; 448 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-26T21:43:04.740 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:04 smithi055 ceph-mon[40587]: pgmap v657: 148 pgs: 148 active+clean; 448 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-26T21:43:04.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:04.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:04.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:05.085 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:05.117 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:05.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:05.249 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:05.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:05.392 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:05.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:05.525 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:05.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:05 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:43:05.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:05 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:43:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:05 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:43:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:05 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:43:05.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:05 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:43:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:05 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:43:05.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:05.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:05.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:05.956 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:05.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:06.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:06.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:06.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:06.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:06.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:06.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:06.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:06.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:06.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:06.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:06.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:06.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: pgmap v658: 148 pgs: 148 active+clean; 332 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T21:43:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:43:06.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:06 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:43:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.759 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:06.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:06.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: pgmap v658: 148 pgs: 148 active+clean; 332 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: pgmap v658: 148 pgs: 148 active+clean; 332 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-26T21:43:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:43:06.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:06 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:43:06.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:06.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:06.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:06.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:07.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:07.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:07.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:07.341 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Removed 130648 objects 2024-03-26T21:43:07.341 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Clean up completed and total clean up time :135.331 2024-03-26T21:43:07.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:07.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:07.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:07.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:07.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:07.680 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-26T21:43:07.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-26T21:43:07.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:07.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:07.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:07.788 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T21:43:07.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:07.856 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:07.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:07.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: 3.8 deep-scrub starts 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: 3.8 deep-scrub ok 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:43:08.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: 3.8 deep-scrub starts 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: 3.8 deep-scrub ok 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:07 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:08.106 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:08.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: 3.8 deep-scrub starts 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: 3.8 deep-scrub ok 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:07 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:08.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:08.357 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:08.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:08.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:08.511 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T21:43:08.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:43:08.583 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:08.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:08.585 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:08.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:08.628 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:08.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:08.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:08.829 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:43:08.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:08.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:08.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:09.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:09.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:09.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: pgmap v659: 148 pgs: 148 active+clean; 215 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: osdmap e323: 8 total, 8 up, 7 in 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3999260484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 4.c deep-scrub starts 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[29406]: 4.c deep-scrub ok 2024-03-26T21:43:09.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: pgmap v659: 148 pgs: 148 active+clean; 215 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: osdmap e323: 8 total, 8 up, 7 in 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3999260484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 4.c deep-scrub starts 2024-03-26T21:43:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:08 smithi055 ceph-mon[40587]: 4.c deep-scrub ok 2024-03-26T21:43:09.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:09.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:09.163 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: pgmap v659: 148 pgs: 148 active+clean; 215 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4051136194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: osdmap e323: 8 total, 8 up, 7 in 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3999260484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 4.c deep-scrub starts 2024-03-26T21:43:09.164 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:08 smithi067 ceph-mon[33346]: 4.c deep-scrub ok 2024-03-26T21:43:09.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:09.252 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:09.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:09.378 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:09.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:09.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:09.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:09.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:09.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:09.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:09.887 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:09.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:09.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:10.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:10.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:09 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:43:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:09 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:43:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:09 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:09 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:43:10.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:09 smithi067 ceph-mon[33346]: osdmap e324: 8 total, 8 up, 7 in 2024-03-26T21:43:10.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[40587]: osdmap e324: 8 total, 8 up, 7 in 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:43:10.292 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:43:10.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:10.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:43:10.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:09 smithi055 ceph-mon[29406]: osdmap e324: 8 total, 8 up, 7 in 2024-03-26T21:43:10.366 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:10.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:10.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:10.515 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:10.617 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:10.639 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:10.642 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:10.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:10.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:10.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:10.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:10.892 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:10.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:10.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:11.010 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:10 smithi055 ceph-mon[40587]: pgmap v662: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:43:11.010 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:10 smithi055 ceph-mon[29406]: pgmap v662: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:43:11.017 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:11.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:11.122 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:10 smithi067 ceph-mon[33346]: pgmap v662: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-26T21:43:11.142 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:11.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:11.267 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:11.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:11.392 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:11.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:11.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:11.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:43:11.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:11.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:11.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:11.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:11.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:11.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:11.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:11.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:12.022 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:12.124 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:12.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:12.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:12.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:12.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:12.402 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:12.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:12.633 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:12.658 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:12.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:12.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:12.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:12.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:13.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:13.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:13.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:13.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:12 smithi067 ceph-mon[33346]: pgmap v663: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 875 op/s 2024-03-26T21:43:13.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:13.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:12 smithi055 ceph-mon[29406]: pgmap v663: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 875 op/s 2024-03-26T21:43:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:12 smithi055 ceph-mon[40587]: pgmap v663: 148 pgs: 148 active+clean; 11 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 875 op/s 2024-03-26T21:43:13.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:13.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:13.566 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:13.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:13.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:13.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:13.824 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:13.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:13.953 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:13.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:14.055 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:14.082 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:14.183 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:14.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:14.311 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:14.338 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:14.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:14.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:14.593 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:14.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:14.823 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:14.847 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:14.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:14.972 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:14.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:15.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:15.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:15.180 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:14 smithi055 ceph-mon[40587]: pgmap v664: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 664 op/s 2024-03-26T21:43:15.180 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:14 smithi055 ceph-mon[29406]: pgmap v664: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 664 op/s 2024-03-26T21:43:15.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:15.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:14 smithi067 ceph-mon[33346]: pgmap v664: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 664 op/s 2024-03-26T21:43:15.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:15.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:15.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:15.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:15.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:15.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:15.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:15.741 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:15.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:15.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:15.882 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_2' removed 2024-03-26T21:43:15.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:16.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:16.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:16.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:16.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3952247663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3952247663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:15 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3952247663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]: dispatch 2024-03-26T21:43:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:15 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T21:43:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:16.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:16.377 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:16.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:16.502 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:16.512 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:43:16.515 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:43:16.604 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:16.628 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:16.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:16.753 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:16.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:16.880 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:16.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:16.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:17.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:17.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: pgmap v665: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 429 op/s 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]': finished 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: osdmap e325: 8 total, 8 up, 7 in 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:16 smithi067 ceph-mon[33346]: osdmap e326: 8 total, 8 up, 7 in 2024-03-26T21:43:17.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: pgmap v665: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 429 op/s 2024-03-26T21:43:17.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:43:17.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-03-26T21:43:17.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]': finished 2024-03-26T21:43:17.256 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: osdmap e325: 8 total, 8 up, 7 in 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[29406]: osdmap e326: 8 total, 8 up, 7 in 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: pgmap v665: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 429 op/s 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [1, 0]}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 0]}]': finished 2024-03-26T21:43:17.257 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T21:43:17.258 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: osdmap e325: 8 total, 8 up, 7 in 2024-03-26T21:43:17.258 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:17.258 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:16 smithi055 ceph-mon[40587]: osdmap e326: 8 total, 8 up, 7 in 2024-03-26T21:43:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.259 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:17.283 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-03-26T21:43:15.872473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2076,"num_read_kb":1924,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":1775781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1001,"ondisk_log_size":1001,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3128060,"kb_used_data":9940,"kb_used_omap":18,"kb_used_meta":3117997,"kb_avail":653088004,"statfs":{"total":671965249536,"available":668762116096,"internally_reserved":0,"allocated":10178560,"data_stored":6282749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19291,"internal_metadata":3192829093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-159514648,"num_objects":-2435,"num_object_clones":0,"num_object_copies":-7305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2435,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-258863,"num_write_kb":-8361473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-41288,"num_bytes_recovered":-2705326104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.069137"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":275,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879794+0000","last_change":"2024-03-26T21:41:48.911741+0000","last_active":"2024-03-26T21:43:09.879794+0000","last_peered":"2024-03-26T21:43:09.879794+0000","last_clean":"2024-03-26T21:43:09.879794+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:43:09.879794+0000","last_undegraded":"2024-03-26T21:43:09.879794+0000","last_fullsized":"2024-03-26T21:43:09.879794+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:48.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:41:48.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831627+0000","last_change":"2024-03-26T21:41:25.610037+0000","last_active":"2024-03-26T21:43:08.831627+0000","last_peered":"2024-03-26T21:43:08.831627+0000","last_clean":"2024-03-26T21:43:08.831627+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:43:08.831627+0000","last_undegraded":"2024-03-26T21:43:08.831627+0000","last_fullsized":"2024-03-26T21:43:08.831627+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:13.967296+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831175+0000","last_change":"2024-03-26T21:40:20.651612+0000","last_active":"2024-03-26T21:43:08.831175+0000","last_peered":"2024-03-26T21:43:08.831175+0000","last_clean":"2024-03-26T21:43:08.831175+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:43:08.831175+0000","last_undegraded":"2024-03-26T21:43:08.831175+0000","last_fullsized":"2024-03-26T21:43:08.831175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.651494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.651494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":286,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879385+0000","last_change":"2024-03-26T21:40:26.944039+0000","last_active":"2024-03-26T21:43:09.879385+0000","last_peered":"2024-03-26T21:43:09.879385+0000","last_clean":"2024-03-26T21:43:09.879385+0000","last_became_active":"2024-03-26T21:36:51.695519+0000","last_became_peered":"2024-03-26T21:36:51.695519+0000","last_unstale":"2024-03-26T21:43:09.879385+0000","last_undegraded":"2024-03-26T21:43:09.879385+0000","last_fullsized":"2024-03-26T21:43:09.879385+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:40:26.943893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:26.943893+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196255+0000","last_change":"2024-03-26T21:41:26.600406+0000","last_active":"2024-03-26T21:43:10.196255+0000","last_peered":"2024-03-26T21:43:10.196255+0000","last_clean":"2024-03-26T21:43:10.196255+0000","last_became_active":"2024-03-26T21:41:26.600082+0000","last_became_peered":"2024-03-26T21:41:26.600082+0000","last_unstale":"2024-03-26T21:43:10.196255+0000","last_undegraded":"2024-03-26T21:43:10.196255+0000","last_fullsized":"2024-03-26T21:43:10.196255+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:20.336684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:41:20.336684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":275,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879263+0000","last_change":"2024-03-26T21:41:38.083671+0000","last_active":"2024-03-26T21:43:09.879263+0000","last_peered":"2024-03-26T21:43:09.879263+0000","last_clean":"2024-03-26T21:43:09.879263+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:43:09.879263+0000","last_undegraded":"2024-03-26T21:43:09.879263+0000","last_fullsized":"2024-03-26T21:43:09.879263+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.083526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":129,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.822023+0000","last_change":"2024-03-26T21:42:57.090990+0000","last_active":"2024-03-26T21:43:08.822023+0000","last_peered":"2024-03-26T21:43:08.822023+0000","last_clean":"2024-03-26T21:43:08.822023+0000","last_became_active":"2024-03-26T21:41:06.689493+0000","last_became_peered":"2024-03-26T21:41:06.689493+0000","last_unstale":"2024-03-26T21:43:08.822023+0000","last_undegraded":"2024-03-26T21:43:08.822023+0000","last_fullsized":"2024-03-26T21:43:08.822023+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:57.090868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.090868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:57.877867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":554,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638306+0000","last_change":"2024-03-26T21:42:43.232904+0000","last_active":"2024-03-26T21:43:09.638306+0000","last_peered":"2024-03-26T21:43:09.638306+0000","last_clean":"2024-03-26T21:43:09.638306+0000","last_became_active":"2024-03-26T21:41:06.687399+0000","last_became_peered":"2024-03-26T21:41:06.687399+0000","last_unstale":"2024-03-26T21:43:09.638306+0000","last_undegraded":"2024-03-26T21:43:09.638306+0000","last_fullsized":"2024-03-26T21:43:09.638306+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:43.232749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:43.232749+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.944023+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":559,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831695+0000","last_change":"2024-03-26T21:41:25.610223+0000","last_active":"2024-03-26T21:43:08.831695+0000","last_peered":"2024-03-26T21:43:08.831695+0000","last_clean":"2024-03-26T21:43:08.831695+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:43:08.831695+0000","last_undegraded":"2024-03-26T21:43:08.831695+0000","last_fullsized":"2024-03-26T21:43:08.831695+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.492302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.492302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170149+0000","last_change":"2024-03-26T21:41:25.611196+0000","last_active":"2024-03-26T21:43:09.170149+0000","last_peered":"2024-03-26T21:43:09.170149+0000","last_clean":"2024-03-26T21:43:09.170149+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:43:09.170149+0000","last_undegraded":"2024-03-26T21:43:09.170149+0000","last_fullsized":"2024-03-26T21:43:09.170149+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.192065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.192065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170696+0000","last_change":"2024-03-26T21:40:42.938835+0000","last_active":"2024-03-26T21:43:09.170696+0000","last_peered":"2024-03-26T21:43:09.170696+0000","last_clean":"2024-03-26T21:43:09.170696+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:43:09.170696+0000","last_undegraded":"2024-03-26T21:43:09.170696+0000","last_fullsized":"2024-03-26T21:43:09.170696+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:42.938743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:40:42.938743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":310,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879985+0000","last_change":"2024-03-26T21:41:53.752959+0000","last_active":"2024-03-26T21:43:09.879985+0000","last_peered":"2024-03-26T21:43:09.879985+0000","last_clean":"2024-03-26T21:43:09.879985+0000","last_became_active":"2024-03-26T21:41:06.682256+0000","last_became_peered":"2024-03-26T21:41:06.682256+0000","last_unstale":"2024-03-26T21:43:09.879985+0000","last_undegraded":"2024-03-26T21:43:09.879985+0000","last_fullsized":"2024-03-26T21:43:09.879985+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:53.752832+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:41:53.752832+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170252+0000","last_change":"2024-03-26T21:41:06.691334+0000","last_active":"2024-03-26T21:43:09.170252+0000","last_peered":"2024-03-26T21:43:09.170252+0000","last_clean":"2024-03-26T21:43:09.170252+0000","last_became_active":"2024-03-26T21:41:06.690326+0000","last_became_peered":"2024-03-26T21:41:06.690326+0000","last_unstale":"2024-03-26T21:43:09.170252+0000","last_undegraded":"2024-03-26T21:43:09.170252+0000","last_fullsized":"2024-03-26T21:43:09.170252+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:48.999826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.999826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":238,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195045+0000","last_change":"2024-03-26T21:41:25.658468+0000","last_active":"2024-03-26T21:43:10.195045+0000","last_peered":"2024-03-26T21:43:10.195045+0000","last_clean":"2024-03-26T21:43:10.195045+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:43:10.195045+0000","last_undegraded":"2024-03-26T21:43:10.195045+0000","last_fullsized":"2024-03-26T21:43:10.195045+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:28.958626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:28.958626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170582+0000","last_change":"2024-03-26T21:40:35.930617+0000","last_active":"2024-03-26T21:43:09.170582+0000","last_peered":"2024-03-26T21:43:09.170582+0000","last_clean":"2024-03-26T21:43:09.170582+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:43:09.170582+0000","last_undegraded":"2024-03-26T21:43:09.170582+0000","last_fullsized":"2024-03-26T21:43:09.170582+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_clean_scrub_stamp":"2024-03-26T21:40:35.930502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":296,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880064+0000","last_change":"2024-03-26T21:43:02.671358+0000","last_active":"2024-03-26T21:43:09.880064+0000","last_peered":"2024-03-26T21:43:09.880064+0000","last_clean":"2024-03-26T21:43:09.880064+0000","last_became_active":"2024-03-26T21:41:06.682153+0000","last_became_peered":"2024-03-26T21:41:06.682153+0000","last_unstale":"2024-03-26T21:43:09.880064+0000","last_undegraded":"2024-03-26T21:43:09.880064+0000","last_fullsized":"2024-03-26T21:43:09.880064+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:43:02.671224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:02.671224+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.142642+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":268,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880980+0000","last_change":"2024-03-26T21:40:40.874687+0000","last_active":"2024-03-26T21:43:09.880980+0000","last_peered":"2024-03-26T21:43:09.880980+0000","last_clean":"2024-03-26T21:43:09.880980+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:43:09.880980+0000","last_undegraded":"2024-03-26T21:43:09.880980+0000","last_fullsized":"2024-03-26T21:43:09.880980+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.874570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.874570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.833091+0000","last_change":"2024-03-26T21:40:16.881228+0000","last_active":"2024-03-26T21:43:08.833091+0000","last_peered":"2024-03-26T21:43:08.833091+0000","last_clean":"2024-03-26T21:43:08.833091+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:43:08.833091+0000","last_undegraded":"2024-03-26T21:43:08.833091+0000","last_fullsized":"2024-03-26T21:43:08.833091+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.881096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":592,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831223+0000","last_change":"2024-03-26T21:42:56.212250+0000","last_active":"2024-03-26T21:43:08.831223+0000","last_peered":"2024-03-26T21:43:08.831223+0000","last_clean":"2024-03-26T21:43:08.831223+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:43:08.831223+0000","last_undegraded":"2024-03-26T21:43:08.831223+0000","last_fullsized":"2024-03-26T21:43:08.831223+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.212112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.563850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":655,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638507+0000","last_change":"2024-03-26T21:42:57.266611+0000","last_active":"2024-03-26T21:43:09.638507+0000","last_peered":"2024-03-26T21:43:09.638507+0000","last_clean":"2024-03-26T21:43:09.638507+0000","last_became_active":"2024-03-26T21:41:06.694407+0000","last_became_peered":"2024-03-26T21:41:06.694407+0000","last_unstale":"2024-03-26T21:43:09.638507+0000","last_undegraded":"2024-03-26T21:43:09.638507+0000","last_fullsized":"2024-03-26T21:43:09.638507+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:57.266450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.266450+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.961671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":138,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.822096+0000","last_change":"2024-03-26T21:42:59.247357+0000","last_active":"2024-03-26T21:43:08.822096+0000","last_peered":"2024-03-26T21:43:08.822096+0000","last_clean":"2024-03-26T21:43:08.822096+0000","last_became_active":"2024-03-26T21:41:06.689834+0000","last_became_peered":"2024-03-26T21:41:06.689834+0000","last_unstale":"2024-03-26T21:43:08.822096+0000","last_undegraded":"2024-03-26T21:43:08.822096+0000","last_fullsized":"2024-03-26T21:43:08.822096+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.247230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.247230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.613979+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":285,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880180+0000","last_change":"2024-03-26T21:41:47.917327+0000","last_active":"2024-03-26T21:43:09.880180+0000","last_peered":"2024-03-26T21:43:09.880180+0000","last_clean":"2024-03-26T21:43:09.880180+0000","last_became_active":"2024-03-26T21:41:26.595687+0000","last_became_peered":"2024-03-26T21:41:26.595687+0000","last_unstale":"2024-03-26T21:43:09.880180+0000","last_undegraded":"2024-03-26T21:43:09.880180+0000","last_fullsized":"2024-03-26T21:43:09.880180+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:47.917202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:41:47.917202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":267,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879702+0000","last_change":"2024-03-26T21:40:29.902688+0000","last_active":"2024-03-26T21:43:09.879702+0000","last_peered":"2024-03-26T21:43:09.879702+0000","last_clean":"2024-03-26T21:43:09.879702+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:43:09.879702+0000","last_undegraded":"2024-03-26T21:43:09.879702+0000","last_fullsized":"2024-03-26T21:43:09.879702+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:29.902606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.902606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":568,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832895+0000","last_change":"2024-03-26T21:41:06.760581+0000","last_active":"2024-03-26T21:43:08.832895+0000","last_peered":"2024-03-26T21:43:08.832895+0000","last_clean":"2024-03-26T21:43:08.832895+0000","last_became_active":"2024-03-26T21:41:06.689846+0000","last_became_peered":"2024-03-26T21:41:06.689846+0000","last_unstale":"2024-03-26T21:43:08.832895+0000","last_undegraded":"2024-03-26T21:43:08.832895+0000","last_fullsized":"2024-03-26T21:43:08.832895+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-26T21:40:38.654652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.654652+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":502,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195517+0000","last_change":"2024-03-26T21:43:06.349697+0000","last_active":"2024-03-26T21:43:10.195517+0000","last_peered":"2024-03-26T21:43:10.195517+0000","last_clean":"2024-03-26T21:43:10.195517+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:43:10.195517+0000","last_undegraded":"2024-03-26T21:43:10.195517+0000","last_fullsized":"2024-03-26T21:43:10.195517+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:06.349429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.349429+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:13.011189+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":801,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.822201+0000","last_change":"2024-03-26T21:41:30.072506+0000","last_active":"2024-03-26T21:43:08.822201+0000","last_peered":"2024-03-26T21:43:08.822201+0000","last_clean":"2024-03-26T21:43:08.822201+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:43:08.822201+0000","last_undegraded":"2024-03-26T21:43:08.822201+0000","last_fullsized":"2024-03-26T21:43:08.822201+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:41:30.072374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:30.072374+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":620,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831865+0000","last_change":"2024-03-26T21:41:06.760645+0000","last_active":"2024-03-26T21:43:08.831865+0000","last_peered":"2024-03-26T21:43:08.831865+0000","last_clean":"2024-03-26T21:43:08.831865+0000","last_became_active":"2024-03-26T21:41:06.699136+0000","last_became_peered":"2024-03-26T21:41:06.699136+0000","last_unstale":"2024-03-26T21:43:08.831865+0000","last_undegraded":"2024-03-26T21:43:08.831865+0000","last_fullsized":"2024-03-26T21:43:08.831865+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:40:23.862904+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.862904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:13.967432+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":615,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831224+0000","last_change":"2024-03-26T21:41:26.631132+0000","last_active":"2024-03-26T21:43:08.831224+0000","last_peered":"2024-03-26T21:43:08.831224+0000","last_clean":"2024-03-26T21:43:08.831224+0000","last_became_active":"2024-03-26T21:41:26.629812+0000","last_became_peered":"2024-03-26T21:41:26.629812+0000","last_unstale":"2024-03-26T21:43:08.831224+0000","last_undegraded":"2024-03-26T21:43:08.831224+0000","last_fullsized":"2024-03-26T21:43:08.831224+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:39:33.305513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.305513+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":587,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832797+0000","last_change":"2024-03-26T21:40:16.025952+0000","last_active":"2024-03-26T21:43:08.832797+0000","last_peered":"2024-03-26T21:43:08.832797+0000","last_clean":"2024-03-26T21:43:08.832797+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:43:08.832797+0000","last_undegraded":"2024-03-26T21:43:08.832797+0000","last_fullsized":"2024-03-26T21:43:08.832797+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.025838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.025838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831516+0000","last_change":"2024-03-26T21:42:59.198607+0000","last_active":"2024-03-26T21:43:08.831516+0000","last_peered":"2024-03-26T21:43:08.831516+0000","last_clean":"2024-03-26T21:43:08.831516+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:43:08.831516+0000","last_undegraded":"2024-03-26T21:43:08.831516+0000","last_fullsized":"2024-03-26T21:43:08.831516+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.198481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.198481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.882421+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":120,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831677+0000","last_change":"2024-03-26T21:41:06.684801+0000","last_active":"2024-03-26T21:43:08.831677+0000","last_peered":"2024-03-26T21:43:08.831677+0000","last_clean":"2024-03-26T21:43:08.831677+0000","last_became_active":"2024-03-26T21:41:06.683627+0000","last_became_peered":"2024-03-26T21:41:06.683627+0000","last_unstale":"2024-03-26T21:43:08.831677+0000","last_undegraded":"2024-03-26T21:43:08.831677+0000","last_fullsized":"2024-03-26T21:43:08.831677+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:24.208179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.208179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":628,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.822205+0000","last_change":"2024-03-26T21:42:49.300575+0000","last_active":"2024-03-26T21:43:08.822205+0000","last_peered":"2024-03-26T21:43:08.822205+0000","last_clean":"2024-03-26T21:43:08.822205+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:43:08.822205+0000","last_undegraded":"2024-03-26T21:43:08.822205+0000","last_fullsized":"2024-03-26T21:43:08.822205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:42:49.300477+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.300477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.397719+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":498,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196137+0000","last_change":"2024-03-26T21:41:25.602252+0000","last_active":"2024-03-26T21:43:10.196137+0000","last_peered":"2024-03-26T21:43:10.196137+0000","last_clean":"2024-03-26T21:43:10.196137+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:43:10.196137+0000","last_undegraded":"2024-03-26T21:43:10.196137+0000","last_fullsized":"2024-03-26T21:43:10.196137+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:54.032885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:54.032885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":609,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821632+0000","last_change":"2024-03-26T21:43:06.168376+0000","last_active":"2024-03-26T21:43:08.821632+0000","last_peered":"2024-03-26T21:43:08.821632+0000","last_clean":"2024-03-26T21:43:08.821632+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:43:08.821632+0000","last_undegraded":"2024-03-26T21:43:08.821632+0000","last_fullsized":"2024-03-26T21:43:08.821632+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.168150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.168150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:11.341635+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831370+0000","last_change":"2024-03-26T21:41:06.674643+0000","last_active":"2024-03-26T21:43:08.831370+0000","last_peered":"2024-03-26T21:43:08.831370+0000","last_clean":"2024-03-26T21:43:08.831370+0000","last_became_active":"2024-03-26T21:41:06.674252+0000","last_became_peered":"2024-03-26T21:41:06.674252+0000","last_unstale":"2024-03-26T21:43:08.831370+0000","last_undegraded":"2024-03-26T21:43:08.831370+0000","last_fullsized":"2024-03-26T21:43:08.831370+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:29.294907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.294907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":513,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831725+0000","last_change":"2024-03-26T21:41:06.726632+0000","last_active":"2024-03-26T21:43:08.831725+0000","last_peered":"2024-03-26T21:43:08.831725+0000","last_clean":"2024-03-26T21:43:08.831725+0000","last_became_active":"2024-03-26T21:41:06.691526+0000","last_became_peered":"2024-03-26T21:41:06.691526+0000","last_unstale":"2024-03-26T21:43:08.831725+0000","last_undegraded":"2024-03-26T21:43:08.831725+0000","last_fullsized":"2024-03-26T21:43:08.831725+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:40:48.743775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.743775+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":121,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830915+0000","last_change":"2024-03-26T21:41:06.682123+0000","last_active":"2024-03-26T21:43:08.830915+0000","last_peered":"2024-03-26T21:43:08.830915+0000","last_clean":"2024-03-26T21:43:08.830915+0000","last_became_active":"2024-03-26T21:41:06.681798+0000","last_became_peered":"2024-03-26T21:41:06.681798+0000","last_unstale":"2024-03-26T21:43:08.830915+0000","last_undegraded":"2024-03-26T21:43:08.830915+0000","last_fullsized":"2024-03-26T21:43:08.830915+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:36.268946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:39:36.268946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":498,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195150+0000","last_change":"2024-03-26T21:42:58.360038+0000","last_active":"2024-03-26T21:43:10.195150+0000","last_peered":"2024-03-26T21:43:10.195150+0000","last_clean":"2024-03-26T21:43:10.195150+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:43:10.195150+0000","last_undegraded":"2024-03-26T21:43:10.195150+0000","last_fullsized":"2024-03-26T21:43:10.195150+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.359906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.359906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.193584+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170715+0000","last_change":"2024-03-26T21:42:50.211297+0000","last_active":"2024-03-26T21:43:09.170715+0000","last_peered":"2024-03-26T21:43:09.170715+0000","last_clean":"2024-03-26T21:43:09.170715+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:43:09.170715+0000","last_undegraded":"2024-03-26T21:43:09.170715+0000","last_fullsized":"2024-03-26T21:43:09.170715+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:50.211214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:42:50.211214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.540886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":503,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196050+0000","last_change":"2024-03-26T21:41:26.595088+0000","last_active":"2024-03-26T21:43:10.196050+0000","last_peered":"2024-03-26T21:43:10.196050+0000","last_clean":"2024-03-26T21:43:10.196050+0000","last_became_active":"2024-03-26T21:41:26.594715+0000","last_became_peered":"2024-03-26T21:41:26.594715+0000","last_unstale":"2024-03-26T21:43:10.196050+0000","last_undegraded":"2024-03-26T21:43:10.196050+0000","last_fullsized":"2024-03-26T21:43:10.196050+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:29.152531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.152531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":559,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830365+0000","last_change":"2024-03-26T21:40:17.751277+0000","last_active":"2024-03-26T21:43:08.830365+0000","last_peered":"2024-03-26T21:43:08.830365+0000","last_clean":"2024-03-26T21:43:08.830365+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:43:08.830365+0000","last_undegraded":"2024-03-26T21:43:08.830365+0000","last_fullsized":"2024-03-26T21:43:08.830365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.748978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.748978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":128,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831030+0000","last_change":"2024-03-26T21:43:00.150777+0000","last_active":"2024-03-26T21:43:08.831030+0000","last_peered":"2024-03-26T21:43:08.831030+0000","last_clean":"2024-03-26T21:43:08.831030+0000","last_became_active":"2024-03-26T21:41:06.669878+0000","last_became_peered":"2024-03-26T21:41:06.669878+0000","last_unstale":"2024-03-26T21:43:08.831030+0000","last_undegraded":"2024-03-26T21:43:08.831030+0000","last_fullsized":"2024-03-26T21:43:08.831030+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.150652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.150652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.910010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":507,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196592+0000","last_change":"2024-03-26T21:41:26.681701+0000","last_active":"2024-03-26T21:43:10.196592+0000","last_peered":"2024-03-26T21:43:10.196592+0000","last_clean":"2024-03-26T21:43:10.196592+0000","last_became_active":"2024-03-26T21:41:26.681323+0000","last_became_peered":"2024-03-26T21:41:26.681323+0000","last_unstale":"2024-03-26T21:43:10.196592+0000","last_undegraded":"2024-03-26T21:43:10.196592+0000","last_fullsized":"2024-03-26T21:43:10.196592+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":604,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832888+0000","last_change":"2024-03-26T21:39:44.903744+0000","last_active":"2024-03-26T21:43:08.832888+0000","last_peered":"2024-03-26T21:43:08.832888+0000","last_clean":"2024-03-26T21:43:08.832888+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:43:08.832888+0000","last_undegraded":"2024-03-26T21:43:08.832888+0000","last_fullsized":"2024-03-26T21:43:08.832888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:44.903549+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:39:44.903549+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":121,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830668+0000","last_change":"2024-03-26T21:41:06.692468+0000","last_active":"2024-03-26T21:43:08.830668+0000","last_peered":"2024-03-26T21:43:08.830668+0000","last_clean":"2024-03-26T21:43:08.830668+0000","last_became_active":"2024-03-26T21:41:06.689995+0000","last_became_peered":"2024-03-26T21:41:06.689995+0000","last_unstale":"2024-03-26T21:43:08.830668+0000","last_undegraded":"2024-03-26T21:43:08.830668+0000","last_fullsized":"2024-03-26T21:43:08.830668+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.810420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":597,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832784+0000","last_change":"2024-03-26T21:41:26.614343+0000","last_active":"2024-03-26T21:43:08.832784+0000","last_peered":"2024-03-26T21:43:08.832784+0000","last_clean":"2024-03-26T21:43:08.832784+0000","last_became_active":"2024-03-26T21:41:26.612029+0000","last_became_peered":"2024-03-26T21:41:26.612029+0000","last_unstale":"2024-03-26T21:43:08.832784+0000","last_undegraded":"2024-03-26T21:43:08.832784+0000","last_fullsized":"2024-03-26T21:43:08.832784+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:36.141522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.141522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":276,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879061+0000","last_change":"2024-03-26T21:41:31.911664+0000","last_active":"2024-03-26T21:43:09.879061+0000","last_peered":"2024-03-26T21:43:09.879061+0000","last_clean":"2024-03-26T21:43:09.879061+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:43:09.879061+0000","last_undegraded":"2024-03-26T21:43:09.879061+0000","last_fullsized":"2024-03-26T21:43:09.879061+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:31.911237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:31.911237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":299,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879943+0000","last_change":"2024-03-26T21:41:44.918112+0000","last_active":"2024-03-26T21:43:09.879943+0000","last_peered":"2024-03-26T21:43:09.879943+0000","last_clean":"2024-03-26T21:43:09.879943+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:43:09.879943+0000","last_undegraded":"2024-03-26T21:43:09.879943+0000","last_fullsized":"2024-03-26T21:43:09.879943+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:44.917970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:41:44.917970+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830764+0000","last_change":"2024-03-26T21:41:26.614022+0000","last_active":"2024-03-26T21:43:08.830764+0000","last_peered":"2024-03-26T21:43:08.830764+0000","last_clean":"2024-03-26T21:43:08.830764+0000","last_became_active":"2024-03-26T21:41:26.607685+0000","last_became_peered":"2024-03-26T21:41:26.607685+0000","last_unstale":"2024-03-26T21:43:08.830764+0000","last_undegraded":"2024-03-26T21:43:08.830764+0000","last_fullsized":"2024-03-26T21:43:08.830764+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:38:40.496984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.496984+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":756,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170906+0000","last_change":"2024-03-26T21:43:08.196513+0000","last_active":"2024-03-26T21:43:09.170906+0000","last_peered":"2024-03-26T21:43:09.170906+0000","last_clean":"2024-03-26T21:43:09.170906+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:43:09.170906+0000","last_undegraded":"2024-03-26T21:43:09.170906+0000","last_fullsized":"2024-03-26T21:43:09.170906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:43:08.196428+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.196428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.141469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":508,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196354+0000","last_change":"2024-03-26T21:41:25.595571+0000","last_active":"2024-03-26T21:43:10.196354+0000","last_peered":"2024-03-26T21:43:10.196354+0000","last_clean":"2024-03-26T21:43:10.196354+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:43:10.196354+0000","last_undegraded":"2024-03-26T21:43:10.196354+0000","last_fullsized":"2024-03-26T21:43:10.196354+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:39.962543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:39.962543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":554,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832651+0000","last_change":"2024-03-26T21:42:58.972473+0000","last_active":"2024-03-26T21:43:08.832651+0000","last_peered":"2024-03-26T21:43:08.832651+0000","last_clean":"2024-03-26T21:43:08.832651+0000","last_became_active":"2024-03-26T21:41:26.612486+0000","last_became_peered":"2024-03-26T21:41:26.612486+0000","last_unstale":"2024-03-26T21:43:08.832651+0000","last_undegraded":"2024-03-26T21:43:08.832651+0000","last_fullsized":"2024-03-26T21:43:08.832651+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:42:58.972320+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.972320+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.479102+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":275,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880838+0000","last_change":"2024-03-26T21:42:54.787575+0000","last_active":"2024-03-26T21:43:09.880838+0000","last_peered":"2024-03-26T21:43:09.880838+0000","last_clean":"2024-03-26T21:43:09.880838+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:43:09.880838+0000","last_undegraded":"2024-03-26T21:43:09.880838+0000","last_fullsized":"2024-03-26T21:43:09.880838+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.787464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.787464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.976476+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"278'12","reported_seq":155,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196687+0000","last_change":"2024-03-26T21:41:26.613897+0000","last_active":"2024-03-26T21:43:10.196687+0000","last_peered":"2024-03-26T21:43:10.196687+0000","last_clean":"2024-03-26T21:43:10.196687+0000","last_became_active":"2024-03-26T21:41:26.612842+0000","last_became_peered":"2024-03-26T21:41:26.612842+0000","last_unstale":"2024-03-26T21:43:10.196687+0000","last_undegraded":"2024-03-26T21:43:10.196687+0000","last_fullsized":"2024-03-26T21:43:10.196687+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:39:27.133215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:27.133215+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830724+0000","last_change":"2024-03-26T21:43:07.208656+0000","last_active":"2024-03-26T21:43:08.830724+0000","last_peered":"2024-03-26T21:43:08.830724+0000","last_clean":"2024-03-26T21:43:08.830724+0000","last_became_active":"2024-03-26T21:41:26.607687+0000","last_became_peered":"2024-03-26T21:41:26.607687+0000","last_unstale":"2024-03-26T21:43:08.830724+0000","last_undegraded":"2024-03-26T21:43:08.830724+0000","last_fullsized":"2024-03-26T21:43:08.830724+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:07.208520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:07.208520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.173304+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":627,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170997+0000","last_change":"2024-03-26T21:42:49.162778+0000","last_active":"2024-03-26T21:43:09.170997+0000","last_peered":"2024-03-26T21:43:09.170997+0000","last_clean":"2024-03-26T21:43:09.170997+0000","last_became_active":"2024-03-26T21:41:26.596451+0000","last_became_peered":"2024-03-26T21:41:26.596451+0000","last_unstale":"2024-03-26T21:43:09.170997+0000","last_undegraded":"2024-03-26T21:43:09.170997+0000","last_fullsized":"2024-03-26T21:43:09.170997+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:49.162638+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.162638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:15.859076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":597,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638707+0000","last_change":"2024-03-26T21:42:44.266704+0000","last_active":"2024-03-26T21:43:09.638707+0000","last_peered":"2024-03-26T21:43:09.638707+0000","last_clean":"2024-03-26T21:43:09.638707+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:43:09.638707+0000","last_undegraded":"2024-03-26T21:43:09.638707+0000","last_fullsized":"2024-03-26T21:43:09.638707+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:44.266582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:44.266582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.439301+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":526,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.833109+0000","last_change":"2024-03-26T21:42:56.932427+0000","last_active":"2024-03-26T21:43:08.833109+0000","last_peered":"2024-03-26T21:43:08.833109+0000","last_clean":"2024-03-26T21:43:08.833109+0000","last_became_active":"2024-03-26T21:41:26.610882+0000","last_became_peered":"2024-03-26T21:41:26.610882+0000","last_unstale":"2024-03-26T21:43:08.833109+0000","last_undegraded":"2024-03-26T21:43:08.833109+0000","last_fullsized":"2024-03-26T21:43:08.833109+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.687295+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":543,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195011+0000","last_change":"2024-03-26T21:41:25.623784+0000","last_active":"2024-03-26T21:43:10.195011+0000","last_peered":"2024-03-26T21:43:10.195011+0000","last_clean":"2024-03-26T21:43:10.195011+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:43:10.195011+0000","last_undegraded":"2024-03-26T21:43:10.195011+0000","last_fullsized":"2024-03-26T21:43:10.195011+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.962120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.962120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":619,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831351+0000","last_change":"2024-03-26T21:42:51.070243+0000","last_active":"2024-03-26T21:43:08.831351+0000","last_peered":"2024-03-26T21:43:08.831351+0000","last_clean":"2024-03-26T21:43:08.831351+0000","last_became_active":"2024-03-26T21:41:26.614253+0000","last_became_peered":"2024-03-26T21:41:26.614253+0000","last_unstale":"2024-03-26T21:43:08.831351+0000","last_undegraded":"2024-03-26T21:43:08.831351+0000","last_fullsized":"2024-03-26T21:43:08.831351+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:51.070126+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.070126+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:00.147180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":498,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195858+0000","last_change":"2024-03-26T21:41:25.622629+0000","last_active":"2024-03-26T21:43:10.195858+0000","last_peered":"2024-03-26T21:43:10.195858+0000","last_clean":"2024-03-26T21:43:10.195858+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:43:10.195858+0000","last_undegraded":"2024-03-26T21:43:10.195858+0000","last_fullsized":"2024-03-26T21:43:10.195858+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.068375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170943+0000","last_change":"2024-03-26T21:41:06.691730+0000","last_active":"2024-03-26T21:43:09.170943+0000","last_peered":"2024-03-26T21:43:09.170943+0000","last_clean":"2024-03-26T21:43:09.170943+0000","last_became_active":"2024-03-26T21:41:06.691045+0000","last_became_peered":"2024-03-26T21:41:06.691045+0000","last_unstale":"2024-03-26T21:43:09.170943+0000","last_undegraded":"2024-03-26T21:43:09.170943+0000","last_fullsized":"2024-03-26T21:43:09.170943+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:34.942806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:34.942806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":587,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821105+0000","last_change":"2024-03-26T21:41:40.089802+0000","last_active":"2024-03-26T21:43:08.821105+0000","last_peered":"2024-03-26T21:43:08.821105+0000","last_clean":"2024-03-26T21:43:08.821105+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:43:08.821105+0000","last_undegraded":"2024-03-26T21:43:08.821105+0000","last_fullsized":"2024-03-26T21:43:08.821105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:40.089721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:40.089721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":512,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196693+0000","last_change":"2024-03-26T21:42:51.267747+0000","last_active":"2024-03-26T21:43:10.196693+0000","last_peered":"2024-03-26T21:43:10.196693+0000","last_clean":"2024-03-26T21:43:10.196693+0000","last_became_active":"2024-03-26T21:41:26.612479+0000","last_became_peered":"2024-03-26T21:41:26.612479+0000","last_unstale":"2024-03-26T21:43:10.196693+0000","last_undegraded":"2024-03-26T21:43:10.196693+0000","last_fullsized":"2024-03-26T21:43:10.196693+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:42:51.267662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.267662+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:10.904667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":559,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170421+0000","last_change":"2024-03-26T21:40:25.046799+0000","last_active":"2024-03-26T21:43:09.170421+0000","last_peered":"2024-03-26T21:43:09.170421+0000","last_clean":"2024-03-26T21:43:09.170421+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:43:09.170421+0000","last_undegraded":"2024-03-26T21:43:09.170421+0000","last_fullsized":"2024-03-26T21:43:09.170421+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:25.046675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.046675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195453+0000","last_change":"2024-03-26T21:41:25.611205+0000","last_active":"2024-03-26T21:43:10.195453+0000","last_peered":"2024-03-26T21:43:10.195453+0000","last_clean":"2024-03-26T21:43:10.195453+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:43:10.195453+0000","last_undegraded":"2024-03-26T21:43:10.195453+0000","last_fullsized":"2024-03-26T21:43:10.195453+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:39:22.057217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:22.057217+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":627,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170123+0000","last_change":"2024-03-26T21:42:52.208110+0000","last_active":"2024-03-26T21:43:09.170123+0000","last_peered":"2024-03-26T21:43:09.170123+0000","last_clean":"2024-03-26T21:43:09.170123+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:43:09.170123+0000","last_undegraded":"2024-03-26T21:43:09.170123+0000","last_fullsized":"2024-03-26T21:43:09.170123+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:52.207964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.207964+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.739899+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":692,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195257+0000","last_change":"2024-03-26T21:41:25.624173+0000","last_active":"2024-03-26T21:43:10.195257+0000","last_peered":"2024-03-26T21:43:10.195257+0000","last_clean":"2024-03-26T21:43:10.195257+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:43:10.195257+0000","last_undegraded":"2024-03-26T21:43:10.195257+0000","last_fullsized":"2024-03-26T21:43:10.195257+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:39:25.023169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.023169+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":114,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195970+0000","last_change":"2024-03-26T21:42:56.446256+0000","last_active":"2024-03-26T21:43:10.195970+0000","last_peered":"2024-03-26T21:43:10.195970+0000","last_clean":"2024-03-26T21:43:10.195970+0000","last_became_active":"2024-03-26T21:41:26.600529+0000","last_became_peered":"2024-03-26T21:41:26.600529+0000","last_unstale":"2024-03-26T21:43:10.195970+0000","last_undegraded":"2024-03-26T21:43:10.195970+0000","last_fullsized":"2024-03-26T21:43:10.195970+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.446156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.446156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.661338+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":553,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195354+0000","last_change":"2024-03-26T21:41:25.610560+0000","last_active":"2024-03-26T21:43:10.195354+0000","last_peered":"2024-03-26T21:43:10.195354+0000","last_clean":"2024-03-26T21:43:10.195354+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:43:10.195354+0000","last_undegraded":"2024-03-26T21:43:10.195354+0000","last_fullsized":"2024-03-26T21:43:10.195354+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:39:32.049629+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:39:32.049629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":311,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879954+0000","last_change":"2024-03-26T21:41:43.970623+0000","last_active":"2024-03-26T21:43:09.879954+0000","last_peered":"2024-03-26T21:43:09.879954+0000","last_clean":"2024-03-26T21:43:09.879954+0000","last_became_active":"2024-03-26T21:41:06.689727+0000","last_became_peered":"2024-03-26T21:41:06.689727+0000","last_unstale":"2024-03-26T21:43:09.879954+0000","last_undegraded":"2024-03-26T21:43:09.879954+0000","last_fullsized":"2024-03-26T21:43:09.879954+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.970479+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":587,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638117+0000","last_change":"2024-03-26T21:42:54.268952+0000","last_active":"2024-03-26T21:43:09.638117+0000","last_peered":"2024-03-26T21:43:09.638117+0000","last_clean":"2024-03-26T21:43:09.638117+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:43:09.638117+0000","last_undegraded":"2024-03-26T21:43:09.638117+0000","last_fullsized":"2024-03-26T21:43:09.638117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.268871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.452361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":490,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195502+0000","last_change":"2024-03-26T21:43:05.373915+0000","last_active":"2024-03-26T21:43:10.195502+0000","last_peered":"2024-03-26T21:43:10.195502+0000","last_clean":"2024-03-26T21:43:10.195502+0000","last_became_active":"2024-03-26T21:41:26.596674+0000","last_became_peered":"2024-03-26T21:41:26.596674+0000","last_unstale":"2024-03-26T21:43:10.195502+0000","last_undegraded":"2024-03-26T21:43:10.195502+0000","last_fullsized":"2024-03-26T21:43:10.195502+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.373726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.373726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.697791+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":595,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638504+0000","last_change":"2024-03-26T21:43:01.366930+0000","last_active":"2024-03-26T21:43:09.638504+0000","last_peered":"2024-03-26T21:43:09.638504+0000","last_clean":"2024-03-26T21:43:09.638504+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:43:09.638504+0000","last_undegraded":"2024-03-26T21:43:09.638504+0000","last_fullsized":"2024-03-26T21:43:09.638504+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.366789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.758642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":523,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195604+0000","last_change":"2024-03-26T21:41:25.626666+0000","last_active":"2024-03-26T21:43:10.195604+0000","last_peered":"2024-03-26T21:43:10.195604+0000","last_clean":"2024-03-26T21:43:10.195604+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:43:10.195604+0000","last_undegraded":"2024-03-26T21:43:10.195604+0000","last_fullsized":"2024-03-26T21:43:10.195604+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.044612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.044612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":656,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821425+0000","last_change":"2024-03-26T21:41:36.095640+0000","last_active":"2024-03-26T21:43:08.821425+0000","last_peered":"2024-03-26T21:43:08.821425+0000","last_clean":"2024-03-26T21:43:08.821425+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:43:08.821425+0000","last_undegraded":"2024-03-26T21:43:08.821425+0000","last_fullsized":"2024-03-26T21:43:08.821425+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:41:36.095502+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:41:36.095502+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830799+0000","last_change":"2024-03-26T21:41:25.611985+0000","last_active":"2024-03-26T21:43:08.830799+0000","last_peered":"2024-03-26T21:43:08.830799+0000","last_clean":"2024-03-26T21:43:08.830799+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:43:08.830799+0000","last_undegraded":"2024-03-26T21:43:08.830799+0000","last_fullsized":"2024-03-26T21:43:08.830799+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:41.522724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:38:41.522724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":606,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638390+0000","last_change":"2024-03-26T21:42:55.277471+0000","last_active":"2024-03-26T21:43:09.638390+0000","last_peered":"2024-03-26T21:43:09.638390+0000","last_clean":"2024-03-26T21:43:09.638390+0000","last_became_active":"2024-03-26T21:41:06.673129+0000","last_became_peered":"2024-03-26T21:41:06.673129+0000","last_unstale":"2024-03-26T21:43:09.638390+0000","last_undegraded":"2024-03-26T21:43:09.638390+0000","last_fullsized":"2024-03-26T21:43:09.638390+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.277361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:12.168390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":585,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170596+0000","last_change":"2024-03-26T21:41:06.690493+0000","last_active":"2024-03-26T21:43:09.170596+0000","last_peered":"2024-03-26T21:43:09.170596+0000","last_clean":"2024-03-26T21:43:09.170596+0000","last_became_active":"2024-03-26T21:41:06.690153+0000","last_became_peered":"2024-03-26T21:41:06.690153+0000","last_unstale":"2024-03-26T21:43:09.170596+0000","last_undegraded":"2024-03-26T21:43:09.170596+0000","last_fullsized":"2024-03-26T21:43:09.170596+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:13.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:40:13.989637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":617,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831423+0000","last_change":"2024-03-26T21:39:46.369958+0000","last_active":"2024-03-26T21:43:08.831423+0000","last_peered":"2024-03-26T21:43:08.831423+0000","last_clean":"2024-03-26T21:43:08.831423+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:43:08.831423+0000","last_undegraded":"2024-03-26T21:43:08.831423+0000","last_fullsized":"2024-03-26T21:43:08.831423+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:39:46.369461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:46.369461+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":544,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880476+0000","last_change":"2024-03-26T21:41:27.901875+0000","last_active":"2024-03-26T21:43:09.880476+0000","last_peered":"2024-03-26T21:43:09.880476+0000","last_clean":"2024-03-26T21:43:09.880476+0000","last_became_active":"2024-03-26T21:41:26.593537+0000","last_became_peered":"2024-03-26T21:41:26.593537+0000","last_unstale":"2024-03-26T21:43:09.880476+0000","last_undegraded":"2024-03-26T21:43:09.880476+0000","last_fullsized":"2024-03-26T21:43:09.880476+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:27.901800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:41:27.901800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170160+0000","last_change":"2024-03-26T21:40:36.901922+0000","last_active":"2024-03-26T21:43:09.170160+0000","last_peered":"2024-03-26T21:43:09.170160+0000","last_clean":"2024-03-26T21:43:09.170160+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:43:09.170160+0000","last_undegraded":"2024-03-26T21:43:09.170160+0000","last_fullsized":"2024-03-26T21:43:09.170160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:40:36.901814+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.901814+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":276,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879840+0000","last_change":"2024-03-26T21:41:45.930676+0000","last_active":"2024-03-26T21:43:09.879840+0000","last_peered":"2024-03-26T21:43:09.879840+0000","last_clean":"2024-03-26T21:43:09.879840+0000","last_became_active":"2024-03-26T21:36:51.695776+0000","last_became_peered":"2024-03-26T21:36:51.695776+0000","last_unstale":"2024-03-26T21:43:09.879840+0000","last_undegraded":"2024-03-26T21:43:09.879840+0000","last_fullsized":"2024-03-26T21:43:09.879840+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.930550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.930550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"270'42","reported_seq":800,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170370+0000","last_change":"2024-03-26T21:42:55.163748+0000","last_active":"2024-03-26T21:43:09.170370+0000","last_peered":"2024-03-26T21:43:09.170370+0000","last_clean":"2024-03-26T21:43:09.170370+0000","last_became_active":"2024-03-26T21:41:06.684878+0000","last_became_peered":"2024-03-26T21:41:06.684878+0000","last_unstale":"2024-03-26T21:43:09.170370+0000","last_undegraded":"2024-03-26T21:43:09.170370+0000","last_fullsized":"2024-03-26T21:43:09.170370+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'42","last_scrub_stamp":"2024-03-26T21:42:55.163584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.163584+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:22.239591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831850+0000","last_change":"2024-03-26T21:40:38.776211+0000","last_active":"2024-03-26T21:43:08.831850+0000","last_peered":"2024-03-26T21:43:08.831850+0000","last_clean":"2024-03-26T21:43:08.831850+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:43:08.831850+0000","last_undegraded":"2024-03-26T21:43:08.831850+0000","last_fullsized":"2024-03-26T21:43:08.831850+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:38.774895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.774895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":482,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196504+0000","last_change":"2024-03-26T21:41:26.597095+0000","last_active":"2024-03-26T21:43:10.196504+0000","last_peered":"2024-03-26T21:43:10.196504+0000","last_clean":"2024-03-26T21:43:10.196504+0000","last_became_active":"2024-03-26T21:41:26.596693+0000","last_became_peered":"2024-03-26T21:41:26.596693+0000","last_unstale":"2024-03-26T21:43:10.196504+0000","last_undegraded":"2024-03-26T21:43:10.196504+0000","last_fullsized":"2024-03-26T21:43:10.196504+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:23.213806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:41:23.213806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":596,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831629+0000","last_change":"2024-03-26T21:40:46.085237+0000","last_active":"2024-03-26T21:43:08.831629+0000","last_peered":"2024-03-26T21:43:08.831629+0000","last_clean":"2024-03-26T21:43:08.831629+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:43:08.831629+0000","last_undegraded":"2024-03-26T21:43:08.831629+0000","last_fullsized":"2024-03-26T21:43:08.831629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:46.085013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:40:46.085013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.638233+0000","last_change":"2024-03-26T21:42:49.297990+0000","last_active":"2024-03-26T21:43:09.638233+0000","last_peered":"2024-03-26T21:43:09.638233+0000","last_clean":"2024-03-26T21:43:09.638233+0000","last_became_active":"2024-03-26T21:41:06.690996+0000","last_became_peered":"2024-03-26T21:41:06.690996+0000","last_unstale":"2024-03-26T21:43:09.638233+0000","last_undegraded":"2024-03-26T21:43:09.638233+0000","last_fullsized":"2024-03-26T21:43:09.638233+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:49.297842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.297842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:50.237836+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":621,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.833214+0000","last_change":"2024-03-26T21:40:17.856788+0000","last_active":"2024-03-26T21:43:08.833214+0000","last_peered":"2024-03-26T21:43:08.833214+0000","last_clean":"2024-03-26T21:43:08.833214+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:43:08.833214+0000","last_undegraded":"2024-03-26T21:43:08.833214+0000","last_fullsized":"2024-03-26T21:43:08.833214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:40:17.856643+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.856643+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.637898+0000","last_change":"2024-03-26T21:42:36.265946+0000","last_active":"2024-03-26T21:43:09.637898+0000","last_peered":"2024-03-26T21:43:09.637898+0000","last_clean":"2024-03-26T21:43:09.637898+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:43:09.637898+0000","last_undegraded":"2024-03-26T21:43:09.637898+0000","last_fullsized":"2024-03-26T21:43:09.637898+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:36.265827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:42:36.265827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:40.794076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":586,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.171122+0000","last_change":"2024-03-26T21:41:06.692795+0000","last_active":"2024-03-26T21:43:09.171122+0000","last_peered":"2024-03-26T21:43:09.171122+0000","last_clean":"2024-03-26T21:43:09.171122+0000","last_became_active":"2024-03-26T21:41:06.691293+0000","last_became_peered":"2024-03-26T21:41:06.691293+0000","last_unstale":"2024-03-26T21:43:09.171122+0000","last_undegraded":"2024-03-26T21:43:09.171122+0000","last_fullsized":"2024-03-26T21:43:09.171122+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:43.984446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:43.984446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":768,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.194971+0000","last_change":"2024-03-26T21:41:25.722685+0000","last_active":"2024-03-26T21:43:10.194971+0000","last_peered":"2024-03-26T21:43:10.194971+0000","last_clean":"2024-03-26T21:43:10.194971+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:43:10.194971+0000","last_undegraded":"2024-03-26T21:43:10.194971+0000","last_fullsized":"2024-03-26T21:43:10.194971+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:39:24.057586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.057586+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":121,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170201+0000","last_change":"2024-03-26T21:41:06.675109+0000","last_active":"2024-03-26T21:43:09.170201+0000","last_peered":"2024-03-26T21:43:09.170201+0000","last_clean":"2024-03-26T21:43:09.170201+0000","last_became_active":"2024-03-26T21:41:06.674713+0000","last_became_peered":"2024-03-26T21:41:06.674713+0000","last_unstale":"2024-03-26T21:43:09.170201+0000","last_undegraded":"2024-03-26T21:43:09.170201+0000","last_fullsized":"2024-03-26T21:43:09.170201+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:40:41.689096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":879,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.066010+0000","last_change":"2024-03-26T21:43:03.102559+0000","last_active":"2024-03-26T21:43:10.066010+0000","last_peered":"2024-03-26T21:43:10.066010+0000","last_clean":"2024-03-26T21:43:10.066010+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:43:10.066010+0000","last_undegraded":"2024-03-26T21:43:10.066010+0000","last_fullsized":"2024-03-26T21:43:10.066010+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:43:03.102463+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:43:03.102463+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:23.082632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":508,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195397+0000","last_change":"2024-03-26T21:41:25.608542+0000","last_active":"2024-03-26T21:43:10.195397+0000","last_peered":"2024-03-26T21:43:10.195397+0000","last_clean":"2024-03-26T21:43:10.195397+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:43:10.195397+0000","last_undegraded":"2024-03-26T21:43:10.195397+0000","last_fullsized":"2024-03-26T21:43:10.195397+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:25.982541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.982541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":581,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832197+0000","last_change":"2024-03-26T21:41:06.698565+0000","last_active":"2024-03-26T21:43:08.832197+0000","last_peered":"2024-03-26T21:43:08.832197+0000","last_clean":"2024-03-26T21:43:08.832197+0000","last_became_active":"2024-03-26T21:41:06.698203+0000","last_became_peered":"2024-03-26T21:41:06.698203+0000","last_unstale":"2024-03-26T21:43:08.832197+0000","last_undegraded":"2024-03-26T21:43:08.832197+0000","last_fullsized":"2024-03-26T21:43:08.832197+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:21.924941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:21.924941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":599,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830677+0000","last_change":"2024-03-26T21:41:06.678442+0000","last_active":"2024-03-26T21:43:08.830677+0000","last_peered":"2024-03-26T21:43:08.830677+0000","last_clean":"2024-03-26T21:43:08.830677+0000","last_became_active":"2024-03-26T21:41:06.676496+0000","last_became_peered":"2024-03-26T21:41:06.676496+0000","last_unstale":"2024-03-26T21:43:08.830677+0000","last_undegraded":"2024-03-26T21:43:08.830677+0000","last_fullsized":"2024-03-26T21:43:08.830677+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:40:47.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1296,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:12.016795+0000","last_change":"2024-03-26T21:41:25.722301+0000","last_active":"2024-03-26T21:43:12.016795+0000","last_peered":"2024-03-26T21:43:12.016795+0000","last_clean":"2024-03-26T21:43:12.016795+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:43:12.016795+0000","last_undegraded":"2024-03-26T21:43:12.016795+0000","last_fullsized":"2024-03-26T21:43:12.016795+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.064714+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":758,"num_read_kb":753,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":623,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.830959+0000","last_change":"2024-03-26T21:40:16.682329+0000","last_active":"2024-03-26T21:43:08.830959+0000","last_peered":"2024-03-26T21:43:08.830959+0000","last_clean":"2024-03-26T21:43:08.830959+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:43:08.830959+0000","last_undegraded":"2024-03-26T21:43:08.830959+0000","last_fullsized":"2024-03-26T21:43:08.830959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:16.682154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.682154+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.171282+0000","last_change":"2024-03-26T21:40:49.926871+0000","last_active":"2024-03-26T21:43:09.171282+0000","last_peered":"2024-03-26T21:43:09.171282+0000","last_clean":"2024-03-26T21:43:09.171282+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:43:09.171282+0000","last_undegraded":"2024-03-26T21:43:09.171282+0000","last_fullsized":"2024-03-26T21:43:09.171282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.926754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.926754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":567,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831675+0000","last_change":"2024-03-26T21:39:39.919957+0000","last_active":"2024-03-26T21:43:08.831675+0000","last_peered":"2024-03-26T21:43:08.831675+0000","last_clean":"2024-03-26T21:43:08.831675+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:43:08.831675+0000","last_undegraded":"2024-03-26T21:43:08.831675+0000","last_fullsized":"2024-03-26T21:43:08.831675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.919791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.919791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":587,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832424+0000","last_change":"2024-03-26T21:40:18.905838+0000","last_active":"2024-03-26T21:43:08.832424+0000","last_peered":"2024-03-26T21:43:08.832424+0000","last_clean":"2024-03-26T21:43:08.832424+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:43:08.832424+0000","last_undegraded":"2024-03-26T21:43:08.832424+0000","last_fullsized":"2024-03-26T21:43:08.832424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:18.905727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:40:18.905727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":628,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831369+0000","last_change":"2024-03-26T21:43:04.127568+0000","last_active":"2024-03-26T21:43:08.831369+0000","last_peered":"2024-03-26T21:43:08.831369+0000","last_clean":"2024-03-26T21:43:08.831369+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:43:08.831369+0000","last_undegraded":"2024-03-26T21:43:08.831369+0000","last_fullsized":"2024-03-26T21:43:08.831369+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:43:04.127429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:04.127429+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.683211+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821441+0000","last_change":"2024-03-26T21:41:45.041708+0000","last_active":"2024-03-26T21:43:08.821441+0000","last_peered":"2024-03-26T21:43:08.821441+0000","last_clean":"2024-03-26T21:43:08.821441+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:43:08.821441+0000","last_undegraded":"2024-03-26T21:43:08.821441+0000","last_fullsized":"2024-03-26T21:43:08.821441+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.041580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.041580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":566,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831473+0000","last_change":"2024-03-26T21:41:06.685939+0000","last_active":"2024-03-26T21:43:08.831473+0000","last_peered":"2024-03-26T21:43:08.831473+0000","last_clean":"2024-03-26T21:43:08.831473+0000","last_became_active":"2024-03-26T21:41:06.683042+0000","last_became_peered":"2024-03-26T21:41:06.683042+0000","last_unstale":"2024-03-26T21:43:08.831473+0000","last_undegraded":"2024-03-26T21:43:08.831473+0000","last_fullsized":"2024-03-26T21:43:08.831473+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:28.770387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:28.770387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":586,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170904+0000","last_change":"2024-03-26T21:41:06.693074+0000","last_active":"2024-03-26T21:43:09.170904+0000","last_peered":"2024-03-26T21:43:09.170904+0000","last_clean":"2024-03-26T21:43:09.170904+0000","last_became_active":"2024-03-26T21:41:06.692529+0000","last_became_peered":"2024-03-26T21:41:06.692529+0000","last_unstale":"2024-03-26T21:43:09.170904+0000","last_undegraded":"2024-03-26T21:43:09.170904+0000","last_fullsized":"2024-03-26T21:43:09.170904+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:37.945472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:40:37.945472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":651,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832898+0000","last_change":"2024-03-26T21:40:29.968002+0000","last_active":"2024-03-26T21:43:08.832898+0000","last_peered":"2024-03-26T21:43:08.832898+0000","last_clean":"2024-03-26T21:43:08.832898+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:43:08.832898+0000","last_undegraded":"2024-03-26T21:43:08.832898+0000","last_fullsized":"2024-03-26T21:43:08.832898+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:40:29.967581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.967581+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:13.967584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":747,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831101+0000","last_change":"2024-03-26T21:40:27.726654+0000","last_active":"2024-03-26T21:43:08.831101+0000","last_peered":"2024-03-26T21:43:08.831101+0000","last_clean":"2024-03-26T21:43:08.831101+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:43:08.831101+0000","last_undegraded":"2024-03-26T21:43:08.831101+0000","last_fullsized":"2024-03-26T21:43:08.831101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:40:27.726492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:27.726492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":491,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195039+0000","last_change":"2024-03-26T21:43:00.377290+0000","last_active":"2024-03-26T21:43:10.195039+0000","last_peered":"2024-03-26T21:43:10.195039+0000","last_clean":"2024-03-26T21:43:10.195039+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:43:10.195039+0000","last_undegraded":"2024-03-26T21:43:10.195039+0000","last_fullsized":"2024-03-26T21:43:10.195039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.377185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.377185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:16.298207+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":585,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831103+0000","last_change":"2024-03-26T21:39:45.352141+0000","last_active":"2024-03-26T21:43:08.831103+0000","last_peered":"2024-03-26T21:43:08.831103+0000","last_clean":"2024-03-26T21:43:08.831103+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:43:08.831103+0000","last_undegraded":"2024-03-26T21:43:08.831103+0000","last_fullsized":"2024-03-26T21:43:08.831103+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:39:45.352017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.352017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":313,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879851+0000","last_change":"2024-03-26T21:42:52.675227+0000","last_active":"2024-03-26T21:43:09.879851+0000","last_peered":"2024-03-26T21:43:09.879851+0000","last_clean":"2024-03-26T21:43:09.879851+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:43:09.879851+0000","last_undegraded":"2024-03-26T21:43:09.879851+0000","last_fullsized":"2024-03-26T21:43:09.879851+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:42:52.674814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.674814+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:01.007259+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"270'13","reported_seq":176,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831515+0000","last_change":"2024-03-26T21:41:06.698496+0000","last_active":"2024-03-26T21:43:08.831515+0000","last_peered":"2024-03-26T21:43:08.831515+0000","last_clean":"2024-03-26T21:43:08.831515+0000","last_became_active":"2024-03-26T21:41:06.690006+0000","last_became_peered":"2024-03-26T21:41:06.690006+0000","last_unstale":"2024-03-26T21:43:08.831515+0000","last_undegraded":"2024-03-26T21:43:08.831515+0000","last_fullsized":"2024-03-26T21:43:08.831515+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:40:23.374040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.374040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:13.967552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":282,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879888+0000","last_change":"2024-03-26T21:41:43.000293+0000","last_active":"2024-03-26T21:43:09.879888+0000","last_peered":"2024-03-26T21:43:09.879888+0000","last_clean":"2024-03-26T21:43:09.879888+0000","last_became_active":"2024-03-26T21:41:06.682029+0000","last_became_peered":"2024-03-26T21:41:06.682029+0000","last_unstale":"2024-03-26T21:43:09.879888+0000","last_undegraded":"2024-03-26T21:43:09.879888+0000","last_fullsized":"2024-03-26T21:43:09.879888+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:43.000172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.000172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":113,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.196584+0000","last_change":"2024-03-26T21:43:08.323733+0000","last_active":"2024-03-26T21:43:10.196584+0000","last_peered":"2024-03-26T21:43:10.196584+0000","last_clean":"2024-03-26T21:43:10.196584+0000","last_became_active":"2024-03-26T21:41:26.606051+0000","last_became_peered":"2024-03-26T21:41:26.606051+0000","last_unstale":"2024-03-26T21:43:10.196584+0000","last_undegraded":"2024-03-26T21:43:10.196584+0000","last_fullsized":"2024-03-26T21:43:10.196584+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:08.323649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.323649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.730481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"317'38","reported_seq":539,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:10.195707+0000","last_change":"2024-03-26T21:42:54.349059+0000","last_active":"2024-03-26T21:43:10.195707+0000","last_peered":"2024-03-26T21:43:10.195707+0000","last_clean":"2024-03-26T21:43:10.195707+0000","last_became_active":"2024-03-26T21:41:26.607985+0000","last_became_peered":"2024-03-26T21:41:26.607985+0000","last_unstale":"2024-03-26T21:43:10.195707+0000","last_undegraded":"2024-03-26T21:43:10.195707+0000","last_fullsized":"2024-03-26T21:43:10.195707+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.348885+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:07.702055+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":281,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880077+0000","last_change":"2024-03-26T21:41:28.893317+0000","last_active":"2024-03-26T21:43:09.880077+0000","last_peered":"2024-03-26T21:43:09.880077+0000","last_clean":"2024-03-26T21:43:09.880077+0000","last_became_active":"2024-03-26T21:41:26.630837+0000","last_became_peered":"2024-03-26T21:41:26.630837+0000","last_unstale":"2024-03-26T21:43:09.880077+0000","last_undegraded":"2024-03-26T21:43:09.880077+0000","last_fullsized":"2024-03-26T21:43:09.880077+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:28.893213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:41:28.893213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":567,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.170689+0000","last_change":"2024-03-26T21:43:01.304736+0000","last_active":"2024-03-26T21:43:09.170689+0000","last_peered":"2024-03-26T21:43:09.170689+0000","last_clean":"2024-03-26T21:43:09.170689+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:43:09.170689+0000","last_undegraded":"2024-03-26T21:43:09.170689+0000","last_fullsized":"2024-03-26T21:43:09.170689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.304625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.304625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.389262+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":356,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880929+0000","last_change":"2024-03-26T21:41:24.943122+0000","last_active":"2024-03-26T21:43:09.880929+0000","last_peered":"2024-03-26T21:43:09.880929+0000","last_clean":"2024-03-26T21:43:09.880929+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:43:09.880929+0000","last_undegraded":"2024-03-26T21:43:09.880929+0000","last_fullsized":"2024-03-26T21:43:09.880929+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:41:24.943030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:24.943030+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":543,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831891+0000","last_change":"2024-03-26T21:43:06.961745+0000","last_active":"2024-03-26T21:43:08.831891+0000","last_peered":"2024-03-26T21:43:08.831891+0000","last_clean":"2024-03-26T21:43:08.831891+0000","last_became_active":"2024-03-26T21:41:26.608779+0000","last_became_peered":"2024-03-26T21:41:26.608779+0000","last_unstale":"2024-03-26T21:43:08.831891+0000","last_undegraded":"2024-03-26T21:43:08.831891+0000","last_fullsized":"2024-03-26T21:43:08.831891+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.961290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:36.730992+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":697,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832940+0000","last_change":"2024-03-26T21:41:26.670298+0000","last_active":"2024-03-26T21:43:08.832940+0000","last_peered":"2024-03-26T21:43:08.832940+0000","last_clean":"2024-03-26T21:43:08.832940+0000","last_became_active":"2024-03-26T21:41:26.613401+0000","last_became_peered":"2024-03-26T21:41:26.613401+0000","last_unstale":"2024-03-26T21:43:08.832940+0000","last_undegraded":"2024-03-26T21:43:08.832940+0000","last_fullsized":"2024-03-26T21:43:08.832940+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:14.873121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:14.873121+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":133,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821822+0000","last_change":"2024-03-26T21:42:53.182450+0000","last_active":"2024-03-26T21:43:08.821822+0000","last_peered":"2024-03-26T21:43:08.821822+0000","last_clean":"2024-03-26T21:43:08.821822+0000","last_became_active":"2024-03-26T21:41:26.594420+0000","last_became_peered":"2024-03-26T21:41:26.594420+0000","last_unstale":"2024-03-26T21:43:08.821822+0000","last_undegraded":"2024-03-26T21:43:08.821822+0000","last_fullsized":"2024-03-26T21:43:08.821822+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:53.182329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:42:53.182329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:54.585869+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.171276+0000","last_change":"2024-03-26T21:43:05.204186+0000","last_active":"2024-03-26T21:43:09.171276+0000","last_peered":"2024-03-26T21:43:09.171276+0000","last_clean":"2024-03-26T21:43:09.171276+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:43:09.171276+0000","last_undegraded":"2024-03-26T21:43:09.171276+0000","last_fullsized":"2024-03-26T21:43:09.171276+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.204079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.204079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.043660+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":276,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.879750+0000","last_change":"2024-03-26T21:41:33.902737+0000","last_active":"2024-03-26T21:43:09.879750+0000","last_peered":"2024-03-26T21:43:09.879750+0000","last_clean":"2024-03-26T21:43:09.879750+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:43:09.879750+0000","last_undegraded":"2024-03-26T21:43:09.879750+0000","last_fullsized":"2024-03-26T21:43:09.879750+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:33.902465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:33.902465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":299,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880786+0000","last_change":"2024-03-26T21:42:57.692467+0000","last_active":"2024-03-26T21:43:09.880786+0000","last_peered":"2024-03-26T21:43:09.880786+0000","last_clean":"2024-03-26T21:43:09.880786+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:43:09.880786+0000","last_undegraded":"2024-03-26T21:43:09.880786+0000","last_fullsized":"2024-03-26T21:43:09.880786+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:57.692111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.692111+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:02.432141+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":602,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.639024+0000","last_change":"2024-03-26T21:42:58.401711+0000","last_active":"2024-03-26T21:43:09.639024+0000","last_peered":"2024-03-26T21:43:09.639024+0000","last_clean":"2024-03-26T21:43:09.639024+0000","last_became_active":"2024-03-26T21:41:06.685057+0000","last_became_peered":"2024-03-26T21:41:06.685057+0000","last_unstale":"2024-03-26T21:43:09.639024+0000","last_undegraded":"2024-03-26T21:43:09.639024+0000","last_fullsized":"2024-03-26T21:43:09.639024+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.401600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.401600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:21.544808+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":600,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.832658+0000","last_change":"2024-03-26T21:40:22.891092+0000","last_active":"2024-03-26T21:43:08.832658+0000","last_peered":"2024-03-26T21:43:08.832658+0000","last_clean":"2024-03-26T21:43:08.832658+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:43:08.832658+0000","last_undegraded":"2024-03-26T21:43:08.832658+0000","last_fullsized":"2024-03-26T21:43:08.832658+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.890287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.890287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":608,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.821259+0000","last_change":"2024-03-26T21:41:38.113058+0000","last_active":"2024-03-26T21:43:08.821259+0000","last_peered":"2024-03-26T21:43:08.821259+0000","last_clean":"2024-03-26T21:43:08.821259+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:43:08.821259+0000","last_undegraded":"2024-03-26T21:43:08.821259+0000","last_fullsized":"2024-03-26T21:43:08.821259+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.112933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.112933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":322,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880476+0000","last_change":"2024-03-26T21:41:50.859131+0000","last_active":"2024-03-26T21:43:09.880476+0000","last_peered":"2024-03-26T21:43:09.880476+0000","last_clean":"2024-03-26T21:43:09.880476+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:43:09.880476+0000","last_undegraded":"2024-03-26T21:43:09.880476+0000","last_fullsized":"2024-03-26T21:43:09.880476+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:41:50.858942+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":275,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880691+0000","last_change":"2024-03-26T21:41:56.818280+0000","last_active":"2024-03-26T21:43:09.880691+0000","last_peered":"2024-03-26T21:43:09.880691+0000","last_clean":"2024-03-26T21:43:09.880691+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:43:09.880691+0000","last_undegraded":"2024-03-26T21:43:09.880691+0000","last_fullsized":"2024-03-26T21:43:09.880691+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:41:56.817883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:11.668825+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":775,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.637928+0000","last_change":"2024-03-26T21:42:45.315356+0000","last_active":"2024-03-26T21:43:09.637928+0000","last_peered":"2024-03-26T21:43:09.637928+0000","last_clean":"2024-03-26T21:43:09.637928+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:43:09.637928+0000","last_undegraded":"2024-03-26T21:43:09.637928+0000","last_fullsized":"2024-03-26T21:43:09.637928+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:42:45.315022+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:42:45.315022+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.152774+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":430,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:08.831355+0000","last_change":"2024-03-26T21:41:06.677479+0000","last_active":"2024-03-26T21:43:08.831355+0000","last_peered":"2024-03-26T21:43:08.831355+0000","last_clean":"2024-03-26T21:43:08.831355+0000","last_became_active":"2024-03-26T21:41:06.675316+0000","last_became_peered":"2024-03-26T21:41:06.675316+0000","last_unstale":"2024-03-26T21:43:08.831355+0000","last_undegraded":"2024-03-26T21:43:08.831355+0000","last_fullsized":"2024-03-26T21:43:08.831355+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.653184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.653184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":320,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-26T21:43:09.880190+0000","last_change":"2024-03-26T21:43:08.600812+0000","last_active":"2024-03-26T21:43:09.880190+0000","last_peered":"2024-03-26T21:43:09.880190+0000","last_clean":"2024-03-26T21:43:09.880190+0000","last_became_active":"2024-03-26T21:41:06.689688+0000","last_became_peered":"2024-03-26T21:41:06.689688+0000","last_unstale":"2024-03-26T21:43:09.880190+0000","last_undegraded":"2024-03-26T21:43:09.880190+0000","last_fullsized":"2024-03-26T21:43:09.880190+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.600649+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:37.688847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":758,"num_read_kb":753,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":4434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973719,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262116,"kb_used_data":1752,"kb_used_omap":6,"kb_used_meta":260345,"kb_avail":93483036,"statfs":{"total":95995035648,"available":95726628864,"internally_reserved":0,"allocated":1794048,"data_stored":1235882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":266593576},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364986,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353644,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93391508,"statfs":{"total":95995035648,"available":95632904192,"internally_reserved":0,"allocated":1798144,"data_stored":1234314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541884,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93203268,"statfs":{"total":95995035648,"available":95440146432,"internally_reserved":0,"allocated":1224704,"data_stored":645075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":506344,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93238808,"statfs":{"total":95995035648,"available":95476539392,"internally_reserved":0,"allocated":1794048,"data_stored":1235188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461316,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93283836,"statfs":{"total":95995035648,"available":95522648064,"internally_reserved":0,"allocated":1171456,"data_stored":643437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149896,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650493,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456744,"kb_used_data":1176,"kb_used_omap":4,"kb_used_meta":455547,"kb_avail":93288408,"statfs":{"total":95995035648,"available":95527329792,"internally_reserved":0,"allocated":1204224,"data_stored":644957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":466480803},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659325,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546012,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199140,"statfs":{"total":95995035648,"available":95435919360,"internally_reserved":0,"allocated":1191936,"data_stored":643896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:43:17.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:43:17.360 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:17.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:17.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:17.508 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:17.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:17.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:17.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:17.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:17.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:17.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:17.993 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:18.017 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:18.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:18.201 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T21:43:18.201 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T21:43:18.202 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-26T21:43:18.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-26T21:43:18.233 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:18.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:18.276 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:18.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:18.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:18.473 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:18.473 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:18.473 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:18.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:18.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:18.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:18.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:18.534 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:18.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:18.660 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:18.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:18.785 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:18.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:18.914 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:18.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:18.998 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:43:19.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:19.019 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:43:19.038 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:43:19.043 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:19.057 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:43:19.079 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:43:19.099 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:43:19.120 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:43:19.141 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:43:19.142 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:43:19.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:19.171 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:19.194 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:43:19.194 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:43:19.194 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:43:19.194 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:43:19.194 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:43:19.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:19.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:19 smithi055 ceph-mon[40587]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1103145277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:19 smithi055 ceph-mon[29406]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1103145277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:19.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:19.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:19 smithi067 ceph-mon[33346]: pgmap v668: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1103145277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:19.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:19.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:19.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:19.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:19.916 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:19.940 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:19.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:20.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:20.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:20.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:20.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:20.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:20.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:20.444 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:20.546 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.570 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:20.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:20.694 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:20.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:20.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:20.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:20.944 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:20.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:21.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:21.070 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:21.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:21.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:21 smithi055 ceph-mon[40587]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:43:21.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:21 smithi055 ceph-mon[29406]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:43:21.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:21.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:21 smithi067 ceph-mon[33346]: pgmap v669: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:43:21.535 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:43:21.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:21.685 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:21.709 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:21.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:21.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:21.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:21.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:21.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:22.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:22.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:22.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:22.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:22.314 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:22.315 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:22.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:22.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:22.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:22.480 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:22.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:22.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:22.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:22.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:22.964 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:22.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:22.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:22.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:23.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:23.114 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:23.186 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:43:23.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:23.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:23.370 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:23.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:23 smithi067 ceph-mon[33346]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:43:23.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:23.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:23 smithi055 ceph-mon[29406]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:43:23.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:23 smithi055 ceph-mon[40587]: pgmap v670: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T21:43:23.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:43:23.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:23.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:23.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:23.853 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:23.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:23.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:23.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:24.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:24.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:43:24.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:24.132 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:24.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:24.257 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:24.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:43:24.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[40587]: osdmap e327: 8 total, 8 up, 7 in 2024-03-26T21:43:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:43:24.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:24 smithi055 ceph-mon[29406]: osdmap e327: 8 total, 8 up, 7 in 2024-03-26T21:43:24.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.388 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:24.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:24.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:24 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:24 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/499254228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:43:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:24 smithi067 ceph-mon[33346]: osdmap e327: 8 total, 8 up, 7 in 2024-03-26T21:43:24.516 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:24.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:24.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:24.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:24.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:24.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:24.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:25.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:25.023 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:25.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:25.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:25.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:25.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:25.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:25 smithi067 ceph-mon[33346]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-26T21:43:25.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:25.532 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:25.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:25 smithi055 ceph-mon[40587]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-26T21:43:25.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:25 smithi055 ceph-mon[29406]: pgmap v672: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 1 objects/s recovering 2024-03-26T21:43:25.611 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:43:25.613 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:43:25.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:25.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:25.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:25.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:25.915 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:25.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:26.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:26.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:26.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:26.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:26.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:26.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:26.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:26.621 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-03-26T21:43:23.608688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2086,"num_read_kb":1934,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1775891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1001,"ondisk_log_size":1001,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3128248,"kb_used_data":10136,"kb_used_omap":18,"kb_used_meta":3117997,"kb_avail":653087816,"statfs":{"total":671965249536,"available":668761923584,"internally_reserved":0,"allocated":10379264,"data_stored":6378390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19291,"internal_metadata":3192829093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.736214"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":279,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787094+0000","last_change":"2024-03-26T21:41:48.911741+0000","last_active":"2024-03-26T21:43:16.787094+0000","last_peered":"2024-03-26T21:43:16.787094+0000","last_clean":"2024-03-26T21:43:16.787094+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:43:16.787094+0000","last_undegraded":"2024-03-26T21:43:16.787094+0000","last_fullsized":"2024-03-26T21:43:16.787094+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:48.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:41:48.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":584,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789208+0000","last_change":"2024-03-26T21:41:25.610037+0000","last_active":"2024-03-26T21:43:16.789208+0000","last_peered":"2024-03-26T21:43:16.789208+0000","last_clean":"2024-03-26T21:43:16.789208+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:43:16.789208+0000","last_undegraded":"2024-03-26T21:43:16.789208+0000","last_fullsized":"2024-03-26T21:43:16.789208+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.855064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.855064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026889+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.796919+0000","last_change":"2024-03-26T21:40:20.651612+0000","last_active":"2024-03-26T21:43:16.796919+0000","last_peered":"2024-03-26T21:43:16.796919+0000","last_clean":"2024-03-26T21:43:16.796919+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:43:16.796919+0000","last_undegraded":"2024-03-26T21:43:16.796919+0000","last_fullsized":"2024-03-26T21:43:16.796919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.651494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.651494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":306,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.802497+0000","last_change":"2024-03-26T21:43:16.802497+0000","last_active":"2024-03-26T21:43:16.802497+0000","last_peered":"2024-03-26T21:43:16.802497+0000","last_clean":"2024-03-26T21:43:16.802497+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:43:16.802497+0000","last_undegraded":"2024-03-26T21:43:16.802497+0000","last_fullsized":"2024-03-26T21:43:16.802497+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:40:26.943893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:26.943893+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":110,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795317+0000","last_change":"2024-03-26T21:41:26.600406+0000","last_active":"2024-03-26T21:43:16.795317+0000","last_peered":"2024-03-26T21:43:16.795317+0000","last_clean":"2024-03-26T21:43:16.795317+0000","last_became_active":"2024-03-26T21:41:26.600082+0000","last_became_peered":"2024-03-26T21:41:26.600082+0000","last_unstale":"2024-03-26T21:43:16.795317+0000","last_undegraded":"2024-03-26T21:43:16.795317+0000","last_fullsized":"2024-03-26T21:43:16.795317+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:20.336684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:41:20.336684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343275+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":279,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786577+0000","last_change":"2024-03-26T21:41:38.083671+0000","last_active":"2024-03-26T21:43:16.786577+0000","last_peered":"2024-03-26T21:43:16.786577+0000","last_clean":"2024-03-26T21:43:16.786577+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:43:16.786577+0000","last_undegraded":"2024-03-26T21:43:16.786577+0000","last_fullsized":"2024-03-26T21:43:16.786577+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.083526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":133,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789652+0000","last_change":"2024-03-26T21:42:57.090990+0000","last_active":"2024-03-26T21:43:16.789652+0000","last_peered":"2024-03-26T21:43:16.789652+0000","last_clean":"2024-03-26T21:43:16.789652+0000","last_became_active":"2024-03-26T21:41:06.689493+0000","last_became_peered":"2024-03-26T21:41:06.689493+0000","last_unstale":"2024-03-26T21:43:16.789652+0000","last_undegraded":"2024-03-26T21:43:16.789652+0000","last_fullsized":"2024-03-26T21:43:16.789652+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:57.090868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.090868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:57.877867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":558,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.792502+0000","last_change":"2024-03-26T21:42:43.232904+0000","last_active":"2024-03-26T21:43:16.792502+0000","last_peered":"2024-03-26T21:43:16.792502+0000","last_clean":"2024-03-26T21:43:16.792502+0000","last_became_active":"2024-03-26T21:41:06.687399+0000","last_became_peered":"2024-03-26T21:41:06.687399+0000","last_unstale":"2024-03-26T21:43:16.792502+0000","last_undegraded":"2024-03-26T21:43:16.792502+0000","last_fullsized":"2024-03-26T21:43:16.792502+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:43.232749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:43.232749+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.944023+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795071+0000","last_change":"2024-03-26T21:41:25.610223+0000","last_active":"2024-03-26T21:43:16.795071+0000","last_peered":"2024-03-26T21:43:16.795071+0000","last_clean":"2024-03-26T21:43:16.795071+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:43:16.795071+0000","last_undegraded":"2024-03-26T21:43:16.795071+0000","last_fullsized":"2024-03-26T21:43:16.795071+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.492302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.492302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:22.125901+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846311+0000","last_change":"2024-03-26T21:41:25.611196+0000","last_active":"2024-03-26T21:43:16.846311+0000","last_peered":"2024-03-26T21:43:16.846311+0000","last_clean":"2024-03-26T21:43:16.846311+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:43:16.846311+0000","last_undegraded":"2024-03-26T21:43:16.846311+0000","last_fullsized":"2024-03-26T21:43:16.846311+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.192065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.192065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102191+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847646+0000","last_change":"2024-03-26T21:40:42.938835+0000","last_active":"2024-03-26T21:43:16.847646+0000","last_peered":"2024-03-26T21:43:16.847646+0000","last_clean":"2024-03-26T21:43:16.847646+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:43:16.847646+0000","last_undegraded":"2024-03-26T21:43:16.847646+0000","last_fullsized":"2024-03-26T21:43:16.847646+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:42.938743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:40:42.938743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":314,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787628+0000","last_change":"2024-03-26T21:41:53.752959+0000","last_active":"2024-03-26T21:43:16.787628+0000","last_peered":"2024-03-26T21:43:16.787628+0000","last_clean":"2024-03-26T21:43:16.787628+0000","last_became_active":"2024-03-26T21:41:06.682256+0000","last_became_peered":"2024-03-26T21:41:06.682256+0000","last_unstale":"2024-03-26T21:43:16.787628+0000","last_undegraded":"2024-03-26T21:43:16.787628+0000","last_fullsized":"2024-03-26T21:43:16.787628+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:53.752832+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:41:53.752832+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846424+0000","last_change":"2024-03-26T21:41:06.691334+0000","last_active":"2024-03-26T21:43:16.846424+0000","last_peered":"2024-03-26T21:43:16.846424+0000","last_clean":"2024-03-26T21:43:16.846424+0000","last_became_active":"2024-03-26T21:41:06.690326+0000","last_became_peered":"2024-03-26T21:41:06.690326+0000","last_unstale":"2024-03-26T21:43:16.846424+0000","last_undegraded":"2024-03-26T21:43:16.846424+0000","last_fullsized":"2024-03-26T21:43:16.846424+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:48.999826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.999826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102547+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":242,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795182+0000","last_change":"2024-03-26T21:41:25.658468+0000","last_active":"2024-03-26T21:43:16.795182+0000","last_peered":"2024-03-26T21:43:16.795182+0000","last_clean":"2024-03-26T21:43:16.795182+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:43:16.795182+0000","last_undegraded":"2024-03-26T21:43:16.795182+0000","last_fullsized":"2024-03-26T21:43:16.795182+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:28.958626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:28.958626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847254+0000","last_change":"2024-03-26T21:40:35.930617+0000","last_active":"2024-03-26T21:43:16.847254+0000","last_peered":"2024-03-26T21:43:16.847254+0000","last_clean":"2024-03-26T21:43:16.847254+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:43:16.847254+0000","last_undegraded":"2024-03-26T21:43:16.847254+0000","last_fullsized":"2024-03-26T21:43:16.847254+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_clean_scrub_stamp":"2024-03-26T21:40:35.930502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102302+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":300,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787510+0000","last_change":"2024-03-26T21:43:02.671358+0000","last_active":"2024-03-26T21:43:16.787510+0000","last_peered":"2024-03-26T21:43:16.787510+0000","last_clean":"2024-03-26T21:43:16.787510+0000","last_became_active":"2024-03-26T21:41:06.682153+0000","last_became_peered":"2024-03-26T21:41:06.682153+0000","last_unstale":"2024-03-26T21:43:16.787510+0000","last_undegraded":"2024-03-26T21:43:16.787510+0000","last_fullsized":"2024-03-26T21:43:16.787510+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:43:02.671224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:02.671224+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.142642+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":272,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787753+0000","last_change":"2024-03-26T21:40:40.874687+0000","last_active":"2024-03-26T21:43:16.787753+0000","last_peered":"2024-03-26T21:43:16.787753+0000","last_clean":"2024-03-26T21:43:16.787753+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:43:16.787753+0000","last_undegraded":"2024-03-26T21:43:16.787753+0000","last_fullsized":"2024-03-26T21:43:16.787753+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.874570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.874570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":595,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790464+0000","last_change":"2024-03-26T21:40:16.881228+0000","last_active":"2024-03-26T21:43:16.790464+0000","last_peered":"2024-03-26T21:43:16.790464+0000","last_clean":"2024-03-26T21:43:16.790464+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:43:16.790464+0000","last_undegraded":"2024-03-26T21:43:16.790464+0000","last_fullsized":"2024-03-26T21:43:16.790464+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.881096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027091+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":596,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795434+0000","last_change":"2024-03-26T21:42:56.212250+0000","last_active":"2024-03-26T21:43:16.795434+0000","last_peered":"2024-03-26T21:43:16.795434+0000","last_clean":"2024-03-26T21:43:16.795434+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:43:16.795434+0000","last_undegraded":"2024-03-26T21:43:16.795434+0000","last_fullsized":"2024-03-26T21:43:16.795434+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.212112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.563850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":659,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.792101+0000","last_change":"2024-03-26T21:42:57.266611+0000","last_active":"2024-03-26T21:43:16.792101+0000","last_peered":"2024-03-26T21:43:16.792101+0000","last_clean":"2024-03-26T21:43:16.792101+0000","last_became_active":"2024-03-26T21:41:06.694407+0000","last_became_peered":"2024-03-26T21:41:06.694407+0000","last_unstale":"2024-03-26T21:43:16.792101+0000","last_undegraded":"2024-03-26T21:43:16.792101+0000","last_fullsized":"2024-03-26T21:43:16.792101+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:57.266450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.266450+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.961671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":142,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789953+0000","last_change":"2024-03-26T21:42:59.247357+0000","last_active":"2024-03-26T21:43:16.789953+0000","last_peered":"2024-03-26T21:43:16.789953+0000","last_clean":"2024-03-26T21:43:16.789953+0000","last_became_active":"2024-03-26T21:41:06.689834+0000","last_became_peered":"2024-03-26T21:41:06.689834+0000","last_unstale":"2024-03-26T21:43:16.789953+0000","last_undegraded":"2024-03-26T21:43:16.789953+0000","last_fullsized":"2024-03-26T21:43:16.789953+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.247230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.247230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.613979+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":289,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787544+0000","last_change":"2024-03-26T21:41:47.917327+0000","last_active":"2024-03-26T21:43:16.787544+0000","last_peered":"2024-03-26T21:43:16.787544+0000","last_clean":"2024-03-26T21:43:16.787544+0000","last_became_active":"2024-03-26T21:41:26.595687+0000","last_became_peered":"2024-03-26T21:41:26.595687+0000","last_unstale":"2024-03-26T21:43:16.787544+0000","last_undegraded":"2024-03-26T21:43:16.787544+0000","last_fullsized":"2024-03-26T21:43:16.787544+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:47.917202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:41:47.917202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":271,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786926+0000","last_change":"2024-03-26T21:40:29.902688+0000","last_active":"2024-03-26T21:43:16.786926+0000","last_peered":"2024-03-26T21:43:16.786926+0000","last_clean":"2024-03-26T21:43:16.786926+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:43:16.786926+0000","last_undegraded":"2024-03-26T21:43:16.786926+0000","last_fullsized":"2024-03-26T21:43:16.786926+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:29.902606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.902606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790221+0000","last_change":"2024-03-26T21:41:06.760581+0000","last_active":"2024-03-26T21:43:16.790221+0000","last_peered":"2024-03-26T21:43:16.790221+0000","last_clean":"2024-03-26T21:43:16.790221+0000","last_became_active":"2024-03-26T21:41:06.689846+0000","last_became_peered":"2024-03-26T21:41:06.689846+0000","last_unstale":"2024-03-26T21:43:16.790221+0000","last_undegraded":"2024-03-26T21:43:16.790221+0000","last_fullsized":"2024-03-26T21:43:16.790221+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-26T21:40:38.654652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.654652+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":506,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795242+0000","last_change":"2024-03-26T21:43:06.349697+0000","last_active":"2024-03-26T21:43:16.795242+0000","last_peered":"2024-03-26T21:43:16.795242+0000","last_clean":"2024-03-26T21:43:16.795242+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:43:16.795242+0000","last_undegraded":"2024-03-26T21:43:16.795242+0000","last_fullsized":"2024-03-26T21:43:16.795242+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:06.349429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.349429+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:13.011189+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":808,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790228+0000","last_change":"2024-03-26T21:41:30.072506+0000","last_active":"2024-03-26T21:43:16.790228+0000","last_peered":"2024-03-26T21:43:16.790228+0000","last_clean":"2024-03-26T21:43:16.790228+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:43:16.790228+0000","last_undegraded":"2024-03-26T21:43:16.790228+0000","last_fullsized":"2024-03-26T21:43:16.790228+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:41:30.072374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:30.072374+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":624,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789368+0000","last_change":"2024-03-26T21:41:06.760645+0000","last_active":"2024-03-26T21:43:16.789368+0000","last_peered":"2024-03-26T21:43:16.789368+0000","last_clean":"2024-03-26T21:43:16.789368+0000","last_became_active":"2024-03-26T21:41:06.699136+0000","last_became_peered":"2024-03-26T21:41:06.699136+0000","last_unstale":"2024-03-26T21:43:16.789368+0000","last_undegraded":"2024-03-26T21:43:16.789368+0000","last_fullsized":"2024-03-26T21:43:16.789368+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:40:23.862904+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.862904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027024+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":637,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.803353+0000","last_change":"2024-03-26T21:43:16.803353+0000","last_active":"2024-03-26T21:43:16.803353+0000","last_peered":"2024-03-26T21:43:16.803353+0000","last_clean":"2024-03-26T21:43:16.803353+0000","last_became_active":"2024-03-26T21:43:16.797672+0000","last_became_peered":"2024-03-26T21:43:16.797672+0000","last_unstale":"2024-03-26T21:43:16.803353+0000","last_undegraded":"2024-03-26T21:43:16.803353+0000","last_fullsized":"2024-03-26T21:43:16.803353+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:39:33.305513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.305513+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790638+0000","last_change":"2024-03-26T21:40:16.025952+0000","last_active":"2024-03-26T21:43:16.790638+0000","last_peered":"2024-03-26T21:43:16.790638+0000","last_clean":"2024-03-26T21:43:16.790638+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:43:16.790638+0000","last_undegraded":"2024-03-26T21:43:16.790638+0000","last_fullsized":"2024-03-26T21:43:16.790638+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.025838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.025838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026940+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795980+0000","last_change":"2024-03-26T21:42:59.198607+0000","last_active":"2024-03-26T21:43:16.795980+0000","last_peered":"2024-03-26T21:43:16.795980+0000","last_clean":"2024-03-26T21:43:16.795980+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:43:16.795980+0000","last_undegraded":"2024-03-26T21:43:16.795980+0000","last_fullsized":"2024-03-26T21:43:16.795980+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.198481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.198481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.882421+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":124,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794973+0000","last_change":"2024-03-26T21:41:06.684801+0000","last_active":"2024-03-26T21:43:16.794973+0000","last_peered":"2024-03-26T21:43:16.794973+0000","last_clean":"2024-03-26T21:43:16.794973+0000","last_became_active":"2024-03-26T21:41:06.683627+0000","last_became_peered":"2024-03-26T21:41:06.683627+0000","last_unstale":"2024-03-26T21:43:16.794973+0000","last_undegraded":"2024-03-26T21:43:16.794973+0000","last_fullsized":"2024-03-26T21:43:16.794973+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:24.208179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.208179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":632,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790122+0000","last_change":"2024-03-26T21:42:49.300575+0000","last_active":"2024-03-26T21:43:16.790122+0000","last_peered":"2024-03-26T21:43:16.790122+0000","last_clean":"2024-03-26T21:43:16.790122+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:43:16.790122+0000","last_undegraded":"2024-03-26T21:43:16.790122+0000","last_fullsized":"2024-03-26T21:43:16.790122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:42:49.300477+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.300477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.397719+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":502,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794961+0000","last_change":"2024-03-26T21:41:25.602252+0000","last_active":"2024-03-26T21:43:16.794961+0000","last_peered":"2024-03-26T21:43:16.794961+0000","last_clean":"2024-03-26T21:43:16.794961+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:43:16.794961+0000","last_undegraded":"2024-03-26T21:43:16.794961+0000","last_fullsized":"2024-03-26T21:43:16.794961+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:54.032885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:54.032885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.342644+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":613,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789385+0000","last_change":"2024-03-26T21:43:06.168376+0000","last_active":"2024-03-26T21:43:16.789385+0000","last_peered":"2024-03-26T21:43:16.789385+0000","last_clean":"2024-03-26T21:43:16.789385+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:43:16.789385+0000","last_undegraded":"2024-03-26T21:43:16.789385+0000","last_fullsized":"2024-03-26T21:43:16.789385+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.168150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.168150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:11.341635+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":581,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.797026+0000","last_change":"2024-03-26T21:41:06.674643+0000","last_active":"2024-03-26T21:43:16.797026+0000","last_peered":"2024-03-26T21:43:16.797026+0000","last_clean":"2024-03-26T21:43:16.797026+0000","last_became_active":"2024-03-26T21:41:06.674252+0000","last_became_peered":"2024-03-26T21:41:06.674252+0000","last_unstale":"2024-03-26T21:43:16.797026+0000","last_undegraded":"2024-03-26T21:43:16.797026+0000","last_fullsized":"2024-03-26T21:43:16.797026+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:29.294907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.294907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":517,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789619+0000","last_change":"2024-03-26T21:41:06.726632+0000","last_active":"2024-03-26T21:43:16.789619+0000","last_peered":"2024-03-26T21:43:16.789619+0000","last_clean":"2024-03-26T21:43:16.789619+0000","last_became_active":"2024-03-26T21:41:06.691526+0000","last_became_peered":"2024-03-26T21:41:06.691526+0000","last_unstale":"2024-03-26T21:43:16.789619+0000","last_undegraded":"2024-03-26T21:43:16.789619+0000","last_fullsized":"2024-03-26T21:43:16.789619+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:40:48.743775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.743775+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027204+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":125,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794009+0000","last_change":"2024-03-26T21:41:06.682123+0000","last_active":"2024-03-26T21:43:16.794009+0000","last_peered":"2024-03-26T21:43:16.794009+0000","last_clean":"2024-03-26T21:43:16.794009+0000","last_became_active":"2024-03-26T21:41:06.681798+0000","last_became_peered":"2024-03-26T21:41:06.681798+0000","last_unstale":"2024-03-26T21:43:16.794009+0000","last_undegraded":"2024-03-26T21:43:16.794009+0000","last_fullsized":"2024-03-26T21:43:16.794009+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:36.268946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:39:36.268946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":502,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795042+0000","last_change":"2024-03-26T21:42:58.360038+0000","last_active":"2024-03-26T21:43:16.795042+0000","last_peered":"2024-03-26T21:43:16.795042+0000","last_clean":"2024-03-26T21:43:16.795042+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:43:16.795042+0000","last_undegraded":"2024-03-26T21:43:16.795042+0000","last_fullsized":"2024-03-26T21:43:16.795042+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.359906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.359906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.193584+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.848467+0000","last_change":"2024-03-26T21:42:50.211297+0000","last_active":"2024-03-26T21:43:16.848467+0000","last_peered":"2024-03-26T21:43:16.848467+0000","last_clean":"2024-03-26T21:43:16.848467+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:43:16.848467+0000","last_undegraded":"2024-03-26T21:43:16.848467+0000","last_fullsized":"2024-03-26T21:43:16.848467+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:50.211214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:42:50.211214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.540886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":507,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795179+0000","last_change":"2024-03-26T21:41:26.595088+0000","last_active":"2024-03-26T21:43:16.795179+0000","last_peered":"2024-03-26T21:43:16.795179+0000","last_clean":"2024-03-26T21:43:16.795179+0000","last_became_active":"2024-03-26T21:41:26.594715+0000","last_became_peered":"2024-03-26T21:41:26.594715+0000","last_unstale":"2024-03-26T21:43:16.795179+0000","last_undegraded":"2024-03-26T21:43:16.795179+0000","last_fullsized":"2024-03-26T21:43:16.795179+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:29.152531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.152531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343244+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":563,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794864+0000","last_change":"2024-03-26T21:40:17.751277+0000","last_active":"2024-03-26T21:43:16.794864+0000","last_peered":"2024-03-26T21:43:16.794864+0000","last_clean":"2024-03-26T21:43:16.794864+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:43:16.794864+0000","last_undegraded":"2024-03-26T21:43:16.794864+0000","last_fullsized":"2024-03-26T21:43:16.794864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.748978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.748978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":132,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795308+0000","last_change":"2024-03-26T21:43:00.150777+0000","last_active":"2024-03-26T21:43:16.795308+0000","last_peered":"2024-03-26T21:43:16.795308+0000","last_clean":"2024-03-26T21:43:16.795308+0000","last_became_active":"2024-03-26T21:41:06.669878+0000","last_became_peered":"2024-03-26T21:41:06.669878+0000","last_unstale":"2024-03-26T21:43:16.795308+0000","last_undegraded":"2024-03-26T21:43:16.795308+0000","last_fullsized":"2024-03-26T21:43:16.795308+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.150652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.150652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.910010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":511,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795795+0000","last_change":"2024-03-26T21:41:26.681701+0000","last_active":"2024-03-26T21:43:16.795795+0000","last_peered":"2024-03-26T21:43:16.795795+0000","last_clean":"2024-03-26T21:43:16.795795+0000","last_became_active":"2024-03-26T21:41:26.681323+0000","last_became_peered":"2024-03-26T21:41:26.681323+0000","last_unstale":"2024-03-26T21:43:16.795795+0000","last_undegraded":"2024-03-26T21:43:16.795795+0000","last_fullsized":"2024-03-26T21:43:16.795795+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.342717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":608,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790096+0000","last_change":"2024-03-26T21:39:44.903744+0000","last_active":"2024-03-26T21:43:16.790096+0000","last_peered":"2024-03-26T21:43:16.790096+0000","last_clean":"2024-03-26T21:43:16.790096+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:43:16.790096+0000","last_undegraded":"2024-03-26T21:43:16.790096+0000","last_fullsized":"2024-03-26T21:43:16.790096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:44.903549+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:39:44.903549+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026908+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":125,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794235+0000","last_change":"2024-03-26T21:41:06.692468+0000","last_active":"2024-03-26T21:43:16.794235+0000","last_peered":"2024-03-26T21:43:16.794235+0000","last_clean":"2024-03-26T21:43:16.794235+0000","last_became_active":"2024-03-26T21:41:06.689995+0000","last_became_peered":"2024-03-26T21:41:06.689995+0000","last_unstale":"2024-03-26T21:43:16.794235+0000","last_undegraded":"2024-03-26T21:43:16.794235+0000","last_fullsized":"2024-03-26T21:43:16.794235+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.810420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":601,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789617+0000","last_change":"2024-03-26T21:41:26.614343+0000","last_active":"2024-03-26T21:43:16.789617+0000","last_peered":"2024-03-26T21:43:16.789617+0000","last_clean":"2024-03-26T21:43:16.789617+0000","last_became_active":"2024-03-26T21:41:26.612029+0000","last_became_peered":"2024-03-26T21:41:26.612029+0000","last_unstale":"2024-03-26T21:43:16.789617+0000","last_undegraded":"2024-03-26T21:43:16.789617+0000","last_fullsized":"2024-03-26T21:43:16.789617+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:36.141522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.141522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027076+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":280,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787221+0000","last_change":"2024-03-26T21:41:31.911664+0000","last_active":"2024-03-26T21:43:16.787221+0000","last_peered":"2024-03-26T21:43:16.787221+0000","last_clean":"2024-03-26T21:43:16.787221+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:43:16.787221+0000","last_undegraded":"2024-03-26T21:43:16.787221+0000","last_fullsized":"2024-03-26T21:43:16.787221+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:31.911237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:31.911237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":303,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787388+0000","last_change":"2024-03-26T21:41:44.918112+0000","last_active":"2024-03-26T21:43:16.787388+0000","last_peered":"2024-03-26T21:43:16.787388+0000","last_clean":"2024-03-26T21:43:16.787388+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:43:16.787388+0000","last_undegraded":"2024-03-26T21:43:16.787388+0000","last_fullsized":"2024-03-26T21:43:16.787388+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:44.917970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:41:44.917970+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":573,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795642+0000","last_change":"2024-03-26T21:41:26.614022+0000","last_active":"2024-03-26T21:43:16.795642+0000","last_peered":"2024-03-26T21:43:16.795642+0000","last_clean":"2024-03-26T21:43:16.795642+0000","last_became_active":"2024-03-26T21:41:26.607685+0000","last_became_peered":"2024-03-26T21:41:26.607685+0000","last_unstale":"2024-03-26T21:43:16.795642+0000","last_undegraded":"2024-03-26T21:43:16.795642+0000","last_fullsized":"2024-03-26T21:43:16.795642+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:38:40.496984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:40.496984+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":761,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847795+0000","last_change":"2024-03-26T21:43:08.196513+0000","last_active":"2024-03-26T21:43:16.847795+0000","last_peered":"2024-03-26T21:43:16.847795+0000","last_clean":"2024-03-26T21:43:16.847795+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:43:16.847795+0000","last_undegraded":"2024-03-26T21:43:16.847795+0000","last_fullsized":"2024-03-26T21:43:16.847795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:43:08.196428+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.196428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.141469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":512,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795347+0000","last_change":"2024-03-26T21:41:25.595571+0000","last_active":"2024-03-26T21:43:16.795347+0000","last_peered":"2024-03-26T21:43:16.795347+0000","last_clean":"2024-03-26T21:43:16.795347+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:43:16.795347+0000","last_undegraded":"2024-03-26T21:43:16.795347+0000","last_fullsized":"2024-03-26T21:43:16.795347+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:39.962543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:39.962543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.342605+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":558,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789937+0000","last_change":"2024-03-26T21:42:58.972473+0000","last_active":"2024-03-26T21:43:16.789937+0000","last_peered":"2024-03-26T21:43:16.789937+0000","last_clean":"2024-03-26T21:43:16.789937+0000","last_became_active":"2024-03-26T21:41:26.612486+0000","last_became_peered":"2024-03-26T21:41:26.612486+0000","last_unstale":"2024-03-26T21:43:16.789937+0000","last_undegraded":"2024-03-26T21:43:16.789937+0000","last_fullsized":"2024-03-26T21:43:16.789937+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:42:58.972320+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.972320+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.479102+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":279,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787899+0000","last_change":"2024-03-26T21:42:54.787575+0000","last_active":"2024-03-26T21:43:16.787899+0000","last_peered":"2024-03-26T21:43:16.787899+0000","last_clean":"2024-03-26T21:43:16.787899+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:43:16.787899+0000","last_undegraded":"2024-03-26T21:43:16.787899+0000","last_fullsized":"2024-03-26T21:43:16.787899+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.787464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.787464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.976476+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"278'12","reported_seq":161,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795649+0000","last_change":"2024-03-26T21:41:26.613897+0000","last_active":"2024-03-26T21:43:16.795649+0000","last_peered":"2024-03-26T21:43:16.795649+0000","last_clean":"2024-03-26T21:43:16.795649+0000","last_became_active":"2024-03-26T21:41:26.612842+0000","last_became_peered":"2024-03-26T21:41:26.612842+0000","last_unstale":"2024-03-26T21:43:16.795649+0000","last_undegraded":"2024-03-26T21:43:16.795649+0000","last_fullsized":"2024-03-26T21:43:16.795649+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:39:27.133215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:27.133215+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794144+0000","last_change":"2024-03-26T21:43:07.208656+0000","last_active":"2024-03-26T21:43:16.794144+0000","last_peered":"2024-03-26T21:43:16.794144+0000","last_clean":"2024-03-26T21:43:16.794144+0000","last_became_active":"2024-03-26T21:41:26.607687+0000","last_became_peered":"2024-03-26T21:41:26.607687+0000","last_unstale":"2024-03-26T21:43:16.794144+0000","last_undegraded":"2024-03-26T21:43:16.794144+0000","last_fullsized":"2024-03-26T21:43:16.794144+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:07.208520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:07.208520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.173304+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":630,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.848238+0000","last_change":"2024-03-26T21:42:49.162778+0000","last_active":"2024-03-26T21:43:16.848238+0000","last_peered":"2024-03-26T21:43:16.848238+0000","last_clean":"2024-03-26T21:43:16.848238+0000","last_became_active":"2024-03-26T21:41:26.596451+0000","last_became_peered":"2024-03-26T21:41:26.596451+0000","last_unstale":"2024-03-26T21:43:16.848238+0000","last_undegraded":"2024-03-26T21:43:16.848238+0000","last_fullsized":"2024-03-26T21:43:16.848238+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:49.162638+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.162638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:15.859076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":601,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.792391+0000","last_change":"2024-03-26T21:42:44.266704+0000","last_active":"2024-03-26T21:43:16.792391+0000","last_peered":"2024-03-26T21:43:16.792391+0000","last_clean":"2024-03-26T21:43:16.792391+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:43:16.792391+0000","last_undegraded":"2024-03-26T21:43:16.792391+0000","last_fullsized":"2024-03-26T21:43:16.792391+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:44.266582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:44.266582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.439301+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":530,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790118+0000","last_change":"2024-03-26T21:42:56.932427+0000","last_active":"2024-03-26T21:43:16.790118+0000","last_peered":"2024-03-26T21:43:16.790118+0000","last_clean":"2024-03-26T21:43:16.790118+0000","last_became_active":"2024-03-26T21:41:26.610882+0000","last_became_peered":"2024-03-26T21:41:26.610882+0000","last_unstale":"2024-03-26T21:43:16.790118+0000","last_undegraded":"2024-03-26T21:43:16.790118+0000","last_fullsized":"2024-03-26T21:43:16.790118+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.687295+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":547,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794981+0000","last_change":"2024-03-26T21:41:25.623784+0000","last_active":"2024-03-26T21:43:16.794981+0000","last_peered":"2024-03-26T21:43:16.794981+0000","last_clean":"2024-03-26T21:43:16.794981+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:43:16.794981+0000","last_undegraded":"2024-03-26T21:43:16.794981+0000","last_fullsized":"2024-03-26T21:43:16.794981+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.962120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.962120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":623,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795908+0000","last_change":"2024-03-26T21:42:51.070243+0000","last_active":"2024-03-26T21:43:16.795908+0000","last_peered":"2024-03-26T21:43:16.795908+0000","last_clean":"2024-03-26T21:43:16.795908+0000","last_became_active":"2024-03-26T21:41:26.614253+0000","last_became_peered":"2024-03-26T21:41:26.614253+0000","last_unstale":"2024-03-26T21:43:16.795908+0000","last_undegraded":"2024-03-26T21:43:16.795908+0000","last_fullsized":"2024-03-26T21:43:16.795908+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:51.070126+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.070126+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:00.147180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":502,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795676+0000","last_change":"2024-03-26T21:41:25.622629+0000","last_active":"2024-03-26T21:43:16.795676+0000","last_peered":"2024-03-26T21:43:16.795676+0000","last_clean":"2024-03-26T21:43:16.795676+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:43:16.795676+0000","last_undegraded":"2024-03-26T21:43:16.795676+0000","last_fullsized":"2024-03-26T21:43:16.795676+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.068375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.342674+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.848792+0000","last_change":"2024-03-26T21:41:06.691730+0000","last_active":"2024-03-26T21:43:16.848792+0000","last_peered":"2024-03-26T21:43:16.848792+0000","last_clean":"2024-03-26T21:43:16.848792+0000","last_became_active":"2024-03-26T21:41:06.691045+0000","last_became_peered":"2024-03-26T21:41:06.691045+0000","last_unstale":"2024-03-26T21:43:16.848792+0000","last_undegraded":"2024-03-26T21:43:16.848792+0000","last_fullsized":"2024-03-26T21:43:16.848792+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:34.942806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:34.942806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102488+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789077+0000","last_change":"2024-03-26T21:41:40.089802+0000","last_active":"2024-03-26T21:43:16.789077+0000","last_peered":"2024-03-26T21:43:16.789077+0000","last_clean":"2024-03-26T21:43:16.789077+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:43:16.789077+0000","last_undegraded":"2024-03-26T21:43:16.789077+0000","last_fullsized":"2024-03-26T21:43:16.789077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:40.089721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:40.089721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":516,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.796034+0000","last_change":"2024-03-26T21:42:51.267747+0000","last_active":"2024-03-26T21:43:16.796034+0000","last_peered":"2024-03-26T21:43:16.796034+0000","last_clean":"2024-03-26T21:43:16.796034+0000","last_became_active":"2024-03-26T21:41:26.612479+0000","last_became_peered":"2024-03-26T21:41:26.612479+0000","last_unstale":"2024-03-26T21:43:16.796034+0000","last_undegraded":"2024-03-26T21:43:16.796034+0000","last_fullsized":"2024-03-26T21:43:16.796034+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:42:51.267662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.267662+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:10.904667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":562,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846858+0000","last_change":"2024-03-26T21:40:25.046799+0000","last_active":"2024-03-26T21:43:16.846858+0000","last_peered":"2024-03-26T21:43:16.846858+0000","last_clean":"2024-03-26T21:43:16.846858+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:43:16.846858+0000","last_undegraded":"2024-03-26T21:43:16.846858+0000","last_fullsized":"2024-03-26T21:43:16.846858+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:25.046675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.046675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102368+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":566,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795947+0000","last_change":"2024-03-26T21:41:25.611205+0000","last_active":"2024-03-26T21:43:16.795947+0000","last_peered":"2024-03-26T21:43:16.795947+0000","last_clean":"2024-03-26T21:43:16.795947+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:43:16.795947+0000","last_undegraded":"2024-03-26T21:43:16.795947+0000","last_fullsized":"2024-03-26T21:43:16.795947+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:39:22.057217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:22.057217+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.342832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":630,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846371+0000","last_change":"2024-03-26T21:42:52.208110+0000","last_active":"2024-03-26T21:43:16.846371+0000","last_peered":"2024-03-26T21:43:16.846371+0000","last_clean":"2024-03-26T21:43:16.846371+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:43:16.846371+0000","last_undegraded":"2024-03-26T21:43:16.846371+0000","last_fullsized":"2024-03-26T21:43:16.846371+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:52.207964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.207964+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.739899+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":698,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794859+0000","last_change":"2024-03-26T21:41:25.624173+0000","last_active":"2024-03-26T21:43:16.794859+0000","last_peered":"2024-03-26T21:43:16.794859+0000","last_clean":"2024-03-26T21:43:16.794859+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:43:16.794859+0000","last_undegraded":"2024-03-26T21:43:16.794859+0000","last_fullsized":"2024-03-26T21:43:16.794859+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:39:25.023169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.023169+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":118,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794981+0000","last_change":"2024-03-26T21:42:56.446256+0000","last_active":"2024-03-26T21:43:16.794981+0000","last_peered":"2024-03-26T21:43:16.794981+0000","last_clean":"2024-03-26T21:43:16.794981+0000","last_became_active":"2024-03-26T21:41:26.600529+0000","last_became_peered":"2024-03-26T21:41:26.600529+0000","last_unstale":"2024-03-26T21:43:16.794981+0000","last_undegraded":"2024-03-26T21:43:16.794981+0000","last_fullsized":"2024-03-26T21:43:16.794981+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.446156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.446156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.661338+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":557,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.796072+0000","last_change":"2024-03-26T21:41:25.610560+0000","last_active":"2024-03-26T21:43:16.796072+0000","last_peered":"2024-03-26T21:43:16.796072+0000","last_clean":"2024-03-26T21:43:16.796072+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:43:16.796072+0000","last_undegraded":"2024-03-26T21:43:16.796072+0000","last_fullsized":"2024-03-26T21:43:16.796072+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:39:32.049629+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:39:32.049629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":315,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786886+0000","last_change":"2024-03-26T21:41:43.970623+0000","last_active":"2024-03-26T21:43:16.786886+0000","last_peered":"2024-03-26T21:43:16.786886+0000","last_clean":"2024-03-26T21:43:16.786886+0000","last_became_active":"2024-03-26T21:41:06.689727+0000","last_became_peered":"2024-03-26T21:41:06.689727+0000","last_unstale":"2024-03-26T21:43:16.786886+0000","last_undegraded":"2024-03-26T21:43:16.786886+0000","last_fullsized":"2024-03-26T21:43:16.786886+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.970479+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":591,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790812+0000","last_change":"2024-03-26T21:42:54.268952+0000","last_active":"2024-03-26T21:43:16.790812+0000","last_peered":"2024-03-26T21:43:16.790812+0000","last_clean":"2024-03-26T21:43:16.790812+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:43:16.790812+0000","last_undegraded":"2024-03-26T21:43:16.790812+0000","last_fullsized":"2024-03-26T21:43:16.790812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.268871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.452361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":494,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794720+0000","last_change":"2024-03-26T21:43:05.373915+0000","last_active":"2024-03-26T21:43:16.794720+0000","last_peered":"2024-03-26T21:43:16.794720+0000","last_clean":"2024-03-26T21:43:16.794720+0000","last_became_active":"2024-03-26T21:41:26.596674+0000","last_became_peered":"2024-03-26T21:41:26.596674+0000","last_unstale":"2024-03-26T21:43:16.794720+0000","last_undegraded":"2024-03-26T21:43:16.794720+0000","last_fullsized":"2024-03-26T21:43:16.794720+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.373726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.373726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.697791+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":599,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.792786+0000","last_change":"2024-03-26T21:43:01.366930+0000","last_active":"2024-03-26T21:43:16.792786+0000","last_peered":"2024-03-26T21:43:16.792786+0000","last_clean":"2024-03-26T21:43:16.792786+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:43:16.792786+0000","last_undegraded":"2024-03-26T21:43:16.792786+0000","last_fullsized":"2024-03-26T21:43:16.792786+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.366789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.758642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":527,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795882+0000","last_change":"2024-03-26T21:41:25.626666+0000","last_active":"2024-03-26T21:43:16.795882+0000","last_peered":"2024-03-26T21:43:16.795882+0000","last_clean":"2024-03-26T21:43:16.795882+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:43:16.795882+0000","last_undegraded":"2024-03-26T21:43:16.795882+0000","last_fullsized":"2024-03-26T21:43:16.795882+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.044612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.044612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":660,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789421+0000","last_change":"2024-03-26T21:41:36.095640+0000","last_active":"2024-03-26T21:43:16.789421+0000","last_peered":"2024-03-26T21:43:16.789421+0000","last_clean":"2024-03-26T21:43:16.789421+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:43:16.789421+0000","last_undegraded":"2024-03-26T21:43:16.789421+0000","last_fullsized":"2024-03-26T21:43:16.789421+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:41:36.095502+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:41:36.095502+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794392+0000","last_change":"2024-03-26T21:41:25.611985+0000","last_active":"2024-03-26T21:43:16.794392+0000","last_peered":"2024-03-26T21:43:16.794392+0000","last_clean":"2024-03-26T21:43:16.794392+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:43:16.794392+0000","last_undegraded":"2024-03-26T21:43:16.794392+0000","last_fullsized":"2024-03-26T21:43:16.794392+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:41.522724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:38:41.522724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":610,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.791659+0000","last_change":"2024-03-26T21:42:55.277471+0000","last_active":"2024-03-26T21:43:16.791659+0000","last_peered":"2024-03-26T21:43:16.791659+0000","last_clean":"2024-03-26T21:43:16.791659+0000","last_became_active":"2024-03-26T21:41:06.673129+0000","last_became_peered":"2024-03-26T21:41:06.673129+0000","last_unstale":"2024-03-26T21:43:16.791659+0000","last_undegraded":"2024-03-26T21:43:16.791659+0000","last_fullsized":"2024-03-26T21:43:16.791659+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.277361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:12.168390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847397+0000","last_change":"2024-03-26T21:41:06.690493+0000","last_active":"2024-03-26T21:43:16.847397+0000","last_peered":"2024-03-26T21:43:16.847397+0000","last_clean":"2024-03-26T21:43:16.847397+0000","last_became_active":"2024-03-26T21:41:06.690153+0000","last_became_peered":"2024-03-26T21:41:06.690153+0000","last_unstale":"2024-03-26T21:43:16.847397+0000","last_undegraded":"2024-03-26T21:43:16.847397+0000","last_fullsized":"2024-03-26T21:43:16.847397+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:13.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:40:13.989637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102270+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":621,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795587+0000","last_change":"2024-03-26T21:39:46.369958+0000","last_active":"2024-03-26T21:43:16.795587+0000","last_peered":"2024-03-26T21:43:16.795587+0000","last_clean":"2024-03-26T21:43:16.795587+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:43:16.795587+0000","last_undegraded":"2024-03-26T21:43:16.795587+0000","last_fullsized":"2024-03-26T21:43:16.795587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:39:46.369461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:46.369461+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":549,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787484+0000","last_change":"2024-03-26T21:41:27.901875+0000","last_active":"2024-03-26T21:43:16.787484+0000","last_peered":"2024-03-26T21:43:16.787484+0000","last_clean":"2024-03-26T21:43:16.787484+0000","last_became_active":"2024-03-26T21:41:26.593537+0000","last_became_peered":"2024-03-26T21:41:26.593537+0000","last_unstale":"2024-03-26T21:43:16.787484+0000","last_undegraded":"2024-03-26T21:43:16.787484+0000","last_fullsized":"2024-03-26T21:43:16.787484+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:27.901800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:41:27.901800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:23.480079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846432+0000","last_change":"2024-03-26T21:40:36.901922+0000","last_active":"2024-03-26T21:43:16.846432+0000","last_peered":"2024-03-26T21:43:16.846432+0000","last_clean":"2024-03-26T21:43:16.846432+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:43:16.846432+0000","last_undegraded":"2024-03-26T21:43:16.846432+0000","last_fullsized":"2024-03-26T21:43:16.846432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:40:36.901814+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.901814+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102339+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.797008+0000","last_change":"2024-03-26T21:43:16.796612+0000","last_active":"2024-03-26T21:43:16.797008+0000","last_peered":"2024-03-26T21:43:16.797008+0000","last_clean":"2024-03-26T21:43:16.797008+0000","last_became_active":"2024-03-26T21:43:16.796261+0000","last_became_peered":"2024-03-26T21:43:16.796261+0000","last_unstale":"2024-03-26T21:43:16.797008+0000","last_undegraded":"2024-03-26T21:43:16.797008+0000","last_fullsized":"2024-03-26T21:43:16.797008+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.930550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.930550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"270'42","reported_seq":805,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846838+0000","last_change":"2024-03-26T21:42:55.163748+0000","last_active":"2024-03-26T21:43:16.846838+0000","last_peered":"2024-03-26T21:43:16.846838+0000","last_clean":"2024-03-26T21:43:16.846838+0000","last_became_active":"2024-03-26T21:41:06.684878+0000","last_became_peered":"2024-03-26T21:41:06.684878+0000","last_unstale":"2024-03-26T21:43:16.846838+0000","last_undegraded":"2024-03-26T21:43:16.846838+0000","last_fullsized":"2024-03-26T21:43:16.846838+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'42","last_scrub_stamp":"2024-03-26T21:42:55.163584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.163584+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:22.239591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795635+0000","last_change":"2024-03-26T21:40:38.776211+0000","last_active":"2024-03-26T21:43:16.795635+0000","last_peered":"2024-03-26T21:43:16.795635+0000","last_clean":"2024-03-26T21:43:16.795635+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:43:16.795635+0000","last_undegraded":"2024-03-26T21:43:16.795635+0000","last_fullsized":"2024-03-26T21:43:16.795635+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:38.774895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.774895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":486,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795671+0000","last_change":"2024-03-26T21:41:26.597095+0000","last_active":"2024-03-26T21:43:16.795671+0000","last_peered":"2024-03-26T21:43:16.795671+0000","last_clean":"2024-03-26T21:43:16.795671+0000","last_became_active":"2024-03-26T21:41:26.596693+0000","last_became_peered":"2024-03-26T21:41:26.596693+0000","last_unstale":"2024-03-26T21:43:16.795671+0000","last_undegraded":"2024-03-26T21:43:16.795671+0000","last_fullsized":"2024-03-26T21:43:16.795671+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:23.213806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:41:23.213806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789352+0000","last_change":"2024-03-26T21:40:46.085237+0000","last_active":"2024-03-26T21:43:16.789352+0000","last_peered":"2024-03-26T21:43:16.789352+0000","last_clean":"2024-03-26T21:43:16.789352+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:43:16.789352+0000","last_undegraded":"2024-03-26T21:43:16.789352+0000","last_fullsized":"2024-03-26T21:43:16.789352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:46.085013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:40:46.085013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027167+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.792407+0000","last_change":"2024-03-26T21:42:49.297990+0000","last_active":"2024-03-26T21:43:16.792407+0000","last_peered":"2024-03-26T21:43:16.792407+0000","last_clean":"2024-03-26T21:43:16.792407+0000","last_became_active":"2024-03-26T21:41:06.690996+0000","last_became_peered":"2024-03-26T21:41:06.690996+0000","last_unstale":"2024-03-26T21:43:16.792407+0000","last_undegraded":"2024-03-26T21:43:16.792407+0000","last_fullsized":"2024-03-26T21:43:16.792407+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:49.297842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.297842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:50.237836+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":625,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.790103+0000","last_change":"2024-03-26T21:40:17.856788+0000","last_active":"2024-03-26T21:43:16.790103+0000","last_peered":"2024-03-26T21:43:16.790103+0000","last_clean":"2024-03-26T21:43:16.790103+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:43:16.790103+0000","last_undegraded":"2024-03-26T21:43:16.790103+0000","last_fullsized":"2024-03-26T21:43:16.790103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:40:17.856643+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.856643+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026974+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":584,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794075+0000","last_change":"2024-03-26T21:42:36.265946+0000","last_active":"2024-03-26T21:43:16.794075+0000","last_peered":"2024-03-26T21:43:16.794075+0000","last_clean":"2024-03-26T21:43:16.794075+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:43:16.794075+0000","last_undegraded":"2024-03-26T21:43:16.794075+0000","last_fullsized":"2024-03-26T21:43:16.794075+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:36.265827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:42:36.265827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:40.794076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":589,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.851936+0000","last_change":"2024-03-26T21:41:06.692795+0000","last_active":"2024-03-26T21:43:16.851936+0000","last_peered":"2024-03-26T21:43:16.851936+0000","last_clean":"2024-03-26T21:43:16.851936+0000","last_became_active":"2024-03-26T21:41:06.691293+0000","last_became_peered":"2024-03-26T21:41:06.691293+0000","last_unstale":"2024-03-26T21:43:16.851936+0000","last_undegraded":"2024-03-26T21:43:16.851936+0000","last_fullsized":"2024-03-26T21:43:16.851936+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:43.984446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:43.984446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102617+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":773,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795242+0000","last_change":"2024-03-26T21:41:25.722685+0000","last_active":"2024-03-26T21:43:16.795242+0000","last_peered":"2024-03-26T21:43:16.795242+0000","last_clean":"2024-03-26T21:43:16.795242+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:43:16.795242+0000","last_undegraded":"2024-03-26T21:43:16.795242+0000","last_fullsized":"2024-03-26T21:43:16.795242+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:39:24.057586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.057586+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":124,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.846265+0000","last_change":"2024-03-26T21:41:06.675109+0000","last_active":"2024-03-26T21:43:16.846265+0000","last_peered":"2024-03-26T21:43:16.846265+0000","last_clean":"2024-03-26T21:43:16.846265+0000","last_became_active":"2024-03-26T21:41:06.674713+0000","last_became_peered":"2024-03-26T21:41:06.674713+0000","last_unstale":"2024-03-26T21:43:16.846265+0000","last_undegraded":"2024-03-26T21:43:16.846265+0000","last_fullsized":"2024-03-26T21:43:16.846265+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:40:41.689096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":885,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789986+0000","last_change":"2024-03-26T21:43:03.102559+0000","last_active":"2024-03-26T21:43:16.789986+0000","last_peered":"2024-03-26T21:43:16.789986+0000","last_clean":"2024-03-26T21:43:16.789986+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:43:16.789986+0000","last_undegraded":"2024-03-26T21:43:16.789986+0000","last_fullsized":"2024-03-26T21:43:16.789986+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:43:03.102463+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:43:03.102463+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:23.082632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":512,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795453+0000","last_change":"2024-03-26T21:41:25.608542+0000","last_active":"2024-03-26T21:43:16.795453+0000","last_peered":"2024-03-26T21:43:16.795453+0000","last_clean":"2024-03-26T21:43:16.795453+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:43:16.795453+0000","last_undegraded":"2024-03-26T21:43:16.795453+0000","last_fullsized":"2024-03-26T21:43:16.795453+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:25.982541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.982541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:19.343011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789756+0000","last_change":"2024-03-26T21:41:06.698565+0000","last_active":"2024-03-26T21:43:16.789756+0000","last_peered":"2024-03-26T21:43:16.789756+0000","last_clean":"2024-03-26T21:43:16.789756+0000","last_became_active":"2024-03-26T21:41:06.698203+0000","last_became_peered":"2024-03-26T21:41:06.698203+0000","last_unstale":"2024-03-26T21:43:16.789756+0000","last_undegraded":"2024-03-26T21:43:16.789756+0000","last_fullsized":"2024-03-26T21:43:16.789756+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:21.924941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:21.924941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":603,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794131+0000","last_change":"2024-03-26T21:41:06.678442+0000","last_active":"2024-03-26T21:43:16.794131+0000","last_peered":"2024-03-26T21:43:16.794131+0000","last_clean":"2024-03-26T21:43:16.794131+0000","last_became_active":"2024-03-26T21:41:06.676496+0000","last_became_peered":"2024-03-26T21:41:06.676496+0000","last_unstale":"2024-03-26T21:43:16.794131+0000","last_undegraded":"2024-03-26T21:43:16.794131+0000","last_fullsized":"2024-03-26T21:43:16.794131+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:40:47.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1311,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:23.033437+0000","last_change":"2024-03-26T21:41:25.722301+0000","last_active":"2024-03-26T21:43:23.033437+0000","last_peered":"2024-03-26T21:43:23.033437+0000","last_clean":"2024-03-26T21:43:23.033437+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:43:23.033437+0000","last_undegraded":"2024-03-26T21:43:23.033437+0000","last_fullsized":"2024-03-26T21:43:23.033437+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.064714+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":768,"num_read_kb":763,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":627,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795785+0000","last_change":"2024-03-26T21:40:16.682329+0000","last_active":"2024-03-26T21:43:16.795785+0000","last_peered":"2024-03-26T21:43:16.795785+0000","last_clean":"2024-03-26T21:43:16.795785+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:43:16.795785+0000","last_undegraded":"2024-03-26T21:43:16.795785+0000","last_fullsized":"2024-03-26T21:43:16.795785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:16.682154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.682154+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.852426+0000","last_change":"2024-03-26T21:40:49.926871+0000","last_active":"2024-03-26T21:43:16.852426+0000","last_peered":"2024-03-26T21:43:16.852426+0000","last_clean":"2024-03-26T21:43:16.852426+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:43:16.852426+0000","last_undegraded":"2024-03-26T21:43:16.852426+0000","last_fullsized":"2024-03-26T21:43:16.852426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.926754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.926754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102458+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":572,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789307+0000","last_change":"2024-03-26T21:39:39.919957+0000","last_active":"2024-03-26T21:43:16.789307+0000","last_peered":"2024-03-26T21:43:16.789307+0000","last_clean":"2024-03-26T21:43:16.789307+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:43:16.789307+0000","last_undegraded":"2024-03-26T21:43:16.789307+0000","last_fullsized":"2024-03-26T21:43:16.789307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.919791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.919791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026923+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":592,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789935+0000","last_change":"2024-03-26T21:40:18.905838+0000","last_active":"2024-03-26T21:43:16.789935+0000","last_peered":"2024-03-26T21:43:16.789935+0000","last_clean":"2024-03-26T21:43:16.789935+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:43:16.789935+0000","last_undegraded":"2024-03-26T21:43:16.789935+0000","last_fullsized":"2024-03-26T21:43:16.789935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:18.905727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:40:18.905727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.026955+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":632,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795475+0000","last_change":"2024-03-26T21:43:04.127568+0000","last_active":"2024-03-26T21:43:16.795475+0000","last_peered":"2024-03-26T21:43:16.795475+0000","last_clean":"2024-03-26T21:43:16.795475+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:43:16.795475+0000","last_undegraded":"2024-03-26T21:43:16.795475+0000","last_fullsized":"2024-03-26T21:43:16.795475+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:43:04.127429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:04.127429+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.683211+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":584,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789428+0000","last_change":"2024-03-26T21:41:45.041708+0000","last_active":"2024-03-26T21:43:16.789428+0000","last_peered":"2024-03-26T21:43:16.789428+0000","last_clean":"2024-03-26T21:43:16.789428+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:43:16.789428+0000","last_undegraded":"2024-03-26T21:43:16.789428+0000","last_fullsized":"2024-03-26T21:43:16.789428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.041580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.041580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":570,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.794882+0000","last_change":"2024-03-26T21:41:06.685939+0000","last_active":"2024-03-26T21:43:16.794882+0000","last_peered":"2024-03-26T21:43:16.794882+0000","last_clean":"2024-03-26T21:43:16.794882+0000","last_became_active":"2024-03-26T21:41:06.683042+0000","last_became_peered":"2024-03-26T21:41:06.683042+0000","last_unstale":"2024-03-26T21:43:16.794882+0000","last_undegraded":"2024-03-26T21:43:16.794882+0000","last_fullsized":"2024-03-26T21:43:16.794882+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:28.770387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:28.770387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":589,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847871+0000","last_change":"2024-03-26T21:41:06.693074+0000","last_active":"2024-03-26T21:43:16.847871+0000","last_peered":"2024-03-26T21:43:16.847871+0000","last_clean":"2024-03-26T21:43:16.847871+0000","last_became_active":"2024-03-26T21:41:06.692529+0000","last_became_peered":"2024-03-26T21:41:06.692529+0000","last_unstale":"2024-03-26T21:43:16.847871+0000","last_undegraded":"2024-03-26T21:43:16.847871+0000","last_fullsized":"2024-03-26T21:43:16.847871+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:37.945472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:40:37.945472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:17.102398+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":655,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789741+0000","last_change":"2024-03-26T21:40:29.968002+0000","last_active":"2024-03-26T21:43:16.789741+0000","last_peered":"2024-03-26T21:43:16.789741+0000","last_clean":"2024-03-26T21:43:16.789741+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:43:16.789741+0000","last_undegraded":"2024-03-26T21:43:16.789741+0000","last_fullsized":"2024-03-26T21:43:16.789741+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:40:29.967581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.967581+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":753,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795415+0000","last_change":"2024-03-26T21:40:27.726654+0000","last_active":"2024-03-26T21:43:16.795415+0000","last_peered":"2024-03-26T21:43:16.795415+0000","last_clean":"2024-03-26T21:43:16.795415+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:43:16.795415+0000","last_undegraded":"2024-03-26T21:43:16.795415+0000","last_fullsized":"2024-03-26T21:43:16.795415+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:40:27.726492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:27.726492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:22.126232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":495,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795228+0000","last_change":"2024-03-26T21:43:00.377290+0000","last_active":"2024-03-26T21:43:16.795228+0000","last_peered":"2024-03-26T21:43:16.795228+0000","last_clean":"2024-03-26T21:43:16.795228+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:43:16.795228+0000","last_undegraded":"2024-03-26T21:43:16.795228+0000","last_fullsized":"2024-03-26T21:43:16.795228+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.377185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.377185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:16.298207+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":589,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795787+0000","last_change":"2024-03-26T21:39:45.352141+0000","last_active":"2024-03-26T21:43:16.795787+0000","last_peered":"2024-03-26T21:43:16.795787+0000","last_clean":"2024-03-26T21:43:16.795787+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:43:16.795787+0000","last_undegraded":"2024-03-26T21:43:16.795787+0000","last_fullsized":"2024-03-26T21:43:16.795787+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:39:45.352017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.352017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":317,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786802+0000","last_change":"2024-03-26T21:42:52.675227+0000","last_active":"2024-03-26T21:43:16.786802+0000","last_peered":"2024-03-26T21:43:16.786802+0000","last_clean":"2024-03-26T21:43:16.786802+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:43:16.786802+0000","last_undegraded":"2024-03-26T21:43:16.786802+0000","last_fullsized":"2024-03-26T21:43:16.786802+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:42:52.674814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.674814+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:01.007259+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"270'13","reported_seq":182,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789298+0000","last_change":"2024-03-26T21:41:06.698496+0000","last_active":"2024-03-26T21:43:16.789298+0000","last_peered":"2024-03-26T21:43:16.789298+0000","last_clean":"2024-03-26T21:43:16.789298+0000","last_became_active":"2024-03-26T21:41:06.690006+0000","last_became_peered":"2024-03-26T21:41:06.690006+0000","last_unstale":"2024-03-26T21:43:16.789298+0000","last_undegraded":"2024-03-26T21:43:16.789298+0000","last_fullsized":"2024-03-26T21:43:16.789298+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:40:23.374040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.374040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":286,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787204+0000","last_change":"2024-03-26T21:41:43.000293+0000","last_active":"2024-03-26T21:43:16.787204+0000","last_peered":"2024-03-26T21:43:16.787204+0000","last_clean":"2024-03-26T21:43:16.787204+0000","last_became_active":"2024-03-26T21:41:06.682029+0000","last_became_peered":"2024-03-26T21:41:06.682029+0000","last_unstale":"2024-03-26T21:43:16.787204+0000","last_undegraded":"2024-03-26T21:43:16.787204+0000","last_fullsized":"2024-03-26T21:43:16.787204+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:43.000172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.000172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":117,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795853+0000","last_change":"2024-03-26T21:43:08.323733+0000","last_active":"2024-03-26T21:43:16.795853+0000","last_peered":"2024-03-26T21:43:16.795853+0000","last_clean":"2024-03-26T21:43:16.795853+0000","last_became_active":"2024-03-26T21:41:26.606051+0000","last_became_peered":"2024-03-26T21:41:26.606051+0000","last_unstale":"2024-03-26T21:43:16.795853+0000","last_undegraded":"2024-03-26T21:43:16.795853+0000","last_fullsized":"2024-03-26T21:43:16.795853+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:08.323649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.323649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.730481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"317'38","reported_seq":543,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795005+0000","last_change":"2024-03-26T21:42:54.349059+0000","last_active":"2024-03-26T21:43:16.795005+0000","last_peered":"2024-03-26T21:43:16.795005+0000","last_clean":"2024-03-26T21:43:16.795005+0000","last_became_active":"2024-03-26T21:41:26.607985+0000","last_became_peered":"2024-03-26T21:41:26.607985+0000","last_unstale":"2024-03-26T21:43:16.795005+0000","last_undegraded":"2024-03-26T21:43:16.795005+0000","last_fullsized":"2024-03-26T21:43:16.795005+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.348885+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:07.702055+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":285,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786922+0000","last_change":"2024-03-26T21:41:28.893317+0000","last_active":"2024-03-26T21:43:16.786922+0000","last_peered":"2024-03-26T21:43:16.786922+0000","last_clean":"2024-03-26T21:43:16.786922+0000","last_became_active":"2024-03-26T21:41:26.630837+0000","last_became_peered":"2024-03-26T21:41:26.630837+0000","last_unstale":"2024-03-26T21:43:16.786922+0000","last_undegraded":"2024-03-26T21:43:16.786922+0000","last_fullsized":"2024-03-26T21:43:16.786922+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:28.893213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:41:28.893213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":570,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.847515+0000","last_change":"2024-03-26T21:43:01.304736+0000","last_active":"2024-03-26T21:43:16.847515+0000","last_peered":"2024-03-26T21:43:16.847515+0000","last_clean":"2024-03-26T21:43:16.847515+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:43:16.847515+0000","last_undegraded":"2024-03-26T21:43:16.847515+0000","last_fullsized":"2024-03-26T21:43:16.847515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.304625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.304625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.389262+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":361,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787545+0000","last_change":"2024-03-26T21:41:24.943122+0000","last_active":"2024-03-26T21:43:16.787545+0000","last_peered":"2024-03-26T21:43:16.787545+0000","last_clean":"2024-03-26T21:43:16.787545+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:43:16.787545+0000","last_undegraded":"2024-03-26T21:43:16.787545+0000","last_fullsized":"2024-03-26T21:43:16.787545+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:41:24.943030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:24.943030+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":548,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789550+0000","last_change":"2024-03-26T21:43:06.961745+0000","last_active":"2024-03-26T21:43:16.789550+0000","last_peered":"2024-03-26T21:43:16.789550+0000","last_clean":"2024-03-26T21:43:16.789550+0000","last_became_active":"2024-03-26T21:41:26.608779+0000","last_became_peered":"2024-03-26T21:41:26.608779+0000","last_unstale":"2024-03-26T21:43:16.789550+0000","last_undegraded":"2024-03-26T21:43:16.789550+0000","last_fullsized":"2024-03-26T21:43:16.789550+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.961290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:36.730992+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":701,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789891+0000","last_change":"2024-03-26T21:41:26.670298+0000","last_active":"2024-03-26T21:43:16.789891+0000","last_peered":"2024-03-26T21:43:16.789891+0000","last_clean":"2024-03-26T21:43:16.789891+0000","last_became_active":"2024-03-26T21:41:26.613401+0000","last_became_peered":"2024-03-26T21:41:26.613401+0000","last_unstale":"2024-03-26T21:43:16.789891+0000","last_undegraded":"2024-03-26T21:43:16.789891+0000","last_fullsized":"2024-03-26T21:43:16.789891+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:14.873121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:14.873121+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":138,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789825+0000","last_change":"2024-03-26T21:42:53.182450+0000","last_active":"2024-03-26T21:43:16.789825+0000","last_peered":"2024-03-26T21:43:16.789825+0000","last_clean":"2024-03-26T21:43:16.789825+0000","last_became_active":"2024-03-26T21:41:26.594420+0000","last_became_peered":"2024-03-26T21:41:26.594420+0000","last_unstale":"2024-03-26T21:43:16.789825+0000","last_undegraded":"2024-03-26T21:43:16.789825+0000","last_fullsized":"2024-03-26T21:43:16.789825+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:53.182329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:42:53.182329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:54.585869+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":587,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.852326+0000","last_change":"2024-03-26T21:43:05.204186+0000","last_active":"2024-03-26T21:43:16.852326+0000","last_peered":"2024-03-26T21:43:16.852326+0000","last_clean":"2024-03-26T21:43:16.852326+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:43:16.852326+0000","last_undegraded":"2024-03-26T21:43:16.852326+0000","last_fullsized":"2024-03-26T21:43:16.852326+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.204079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.204079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.043660+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":280,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.786767+0000","last_change":"2024-03-26T21:41:33.902737+0000","last_active":"2024-03-26T21:43:16.786767+0000","last_peered":"2024-03-26T21:43:16.786767+0000","last_clean":"2024-03-26T21:43:16.786767+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:43:16.786767+0000","last_undegraded":"2024-03-26T21:43:16.786767+0000","last_fullsized":"2024-03-26T21:43:16.786767+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:33.902465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:33.902465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":303,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787520+0000","last_change":"2024-03-26T21:42:57.692467+0000","last_active":"2024-03-26T21:43:16.787520+0000","last_peered":"2024-03-26T21:43:16.787520+0000","last_clean":"2024-03-26T21:43:16.787520+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:43:16.787520+0000","last_undegraded":"2024-03-26T21:43:16.787520+0000","last_fullsized":"2024-03-26T21:43:16.787520+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:57.692111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.692111+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:02.432141+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":606,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.793101+0000","last_change":"2024-03-26T21:42:58.401711+0000","last_active":"2024-03-26T21:43:16.793101+0000","last_peered":"2024-03-26T21:43:16.793101+0000","last_clean":"2024-03-26T21:43:16.793101+0000","last_became_active":"2024-03-26T21:41:06.685057+0000","last_became_peered":"2024-03-26T21:41:06.685057+0000","last_unstale":"2024-03-26T21:43:16.793101+0000","last_undegraded":"2024-03-26T21:43:16.793101+0000","last_fullsized":"2024-03-26T21:43:16.793101+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.401600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.401600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:21.544808+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":604,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789989+0000","last_change":"2024-03-26T21:40:22.891092+0000","last_active":"2024-03-26T21:43:16.789989+0000","last_peered":"2024-03-26T21:43:16.789989+0000","last_clean":"2024-03-26T21:43:16.789989+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:43:16.789989+0000","last_undegraded":"2024-03-26T21:43:16.789989+0000","last_fullsized":"2024-03-26T21:43:16.789989+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.890287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.890287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:20.027040+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":612,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.789236+0000","last_change":"2024-03-26T21:41:38.113058+0000","last_active":"2024-03-26T21:43:16.789236+0000","last_peered":"2024-03-26T21:43:16.789236+0000","last_clean":"2024-03-26T21:43:16.789236+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:43:16.789236+0000","last_undegraded":"2024-03-26T21:43:16.789236+0000","last_fullsized":"2024-03-26T21:43:16.789236+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.112933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.112933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":326,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787285+0000","last_change":"2024-03-26T21:41:50.859131+0000","last_active":"2024-03-26T21:43:16.787285+0000","last_peered":"2024-03-26T21:43:16.787285+0000","last_clean":"2024-03-26T21:43:16.787285+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:43:16.787285+0000","last_undegraded":"2024-03-26T21:43:16.787285+0000","last_fullsized":"2024-03-26T21:43:16.787285+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:41:50.858942+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":279,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.787408+0000","last_change":"2024-03-26T21:41:56.818280+0000","last_active":"2024-03-26T21:43:16.787408+0000","last_peered":"2024-03-26T21:43:16.787408+0000","last_clean":"2024-03-26T21:43:16.787408+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:43:16.787408+0000","last_undegraded":"2024-03-26T21:43:16.787408+0000","last_fullsized":"2024-03-26T21:43:16.787408+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:41:56.817883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:21.510797+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":781,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.791675+0000","last_change":"2024-03-26T21:42:45.315356+0000","last_active":"2024-03-26T21:43:16.791675+0000","last_peered":"2024-03-26T21:43:16.791675+0000","last_clean":"2024-03-26T21:43:16.791675+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:43:16.791675+0000","last_undegraded":"2024-03-26T21:43:16.791675+0000","last_fullsized":"2024-03-26T21:43:16.791675+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:42:45.315022+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:42:45.315022+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.152774+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":434,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.795561+0000","last_change":"2024-03-26T21:41:06.677479+0000","last_active":"2024-03-26T21:43:16.795561+0000","last_peered":"2024-03-26T21:43:16.795561+0000","last_clean":"2024-03-26T21:43:16.795561+0000","last_became_active":"2024-03-26T21:41:06.675316+0000","last_became_peered":"2024-03-26T21:41:06.675316+0000","last_unstale":"2024-03-26T21:43:16.795561+0000","last_undegraded":"2024-03-26T21:43:16.795561+0000","last_fullsized":"2024-03-26T21:43:16.795561+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.653184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.653184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":342,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-26T21:43:16.802920+0000","last_change":"2024-03-26T21:43:16.802920+0000","last_active":"2024-03-26T21:43:16.802920+0000","last_peered":"2024-03-26T21:43:16.802920+0000","last_clean":"2024-03-26T21:43:16.802920+0000","last_became_active":"2024-03-26T21:43:16.796404+0000","last_became_peered":"2024-03-26T21:43:16.796404+0000","last_unstale":"2024-03-26T21:43:16.802920+0000","last_undegraded":"2024-03-26T21:43:16.802920+0000","last_fullsized":"2024-03-26T21:43:16.802920+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.600649+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:15.962250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":768,"num_read_kb":763,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973721,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262148,"kb_used_data":1780,"kb_used_omap":6,"kb_used_meta":260345,"kb_avail":93483004,"statfs":{"total":95995035648,"available":95726596096,"internally_reserved":0,"allocated":1822720,"data_stored":1249545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":266593576},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364988,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353668,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93391484,"statfs":{"total":95995035648,"available":95632879616,"internally_reserved":0,"allocated":1826816,"data_stored":1247977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541904,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93203248,"statfs":{"total":95995035648,"available":95440125952,"internally_reserved":0,"allocated":1249280,"data_stored":658628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":506376,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93238776,"statfs":{"total":95995035648,"available":95476506624,"internally_reserved":0,"allocated":1826816,"data_stored":1248961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411536,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461348,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93283804,"statfs":{"total":95995035648,"available":95522615296,"internally_reserved":0,"allocated":1200128,"data_stored":657100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650494,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456768,"kb_used_data":1204,"kb_used_omap":4,"kb_used_meta":455547,"kb_avail":93288384,"statfs":{"total":95995035648,"available":95527305216,"internally_reserved":0,"allocated":1232896,"data_stored":658620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":466480803},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659327,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546036,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199116,"statfs":{"total":95995035648,"available":95435894784,"internally_reserved":0,"allocated":1220608,"data_stored":657559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:43:26.626 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T21:43:26.626 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:43:26.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:26.690 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:26.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:26.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:26.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:26.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:27.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:27.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:27.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:27.200 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:27.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:27.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:27.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:27 smithi067 ceph-mon[33346]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:27 smithi067 ceph-mon[33346]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 12 B/s, 1 objects/s recovering 2024-03-26T21:43:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:27 smithi067 ceph-mon[33346]: osdmap e328: 8 total, 8 up, 7 in 2024-03-26T21:43:27.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:27.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[29406]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:27.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[29406]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 12 B/s, 1 objects/s recovering 2024-03-26T21:43:27.555 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[29406]: osdmap e328: 8 total, 8 up, 7 in 2024-03-26T21:43:27.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[40587]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:27.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[40587]: pgmap v673: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 12 B/s, 1 objects/s recovering 2024-03-26T21:43:27.556 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:27 smithi055 ceph-mon[40587]: osdmap e328: 8 total, 8 up, 7 in 2024-03-26T21:43:27.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:27.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:27.817 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_3' created 2024-03-26T21:43:27.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:27.846 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:27.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:27.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:43:27.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:28.011 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:28.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:28.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:28.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:28.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:28.391 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:43:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:43:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/62763886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T21:43:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:28 smithi067 ceph-mon[33346]: osdmap e329: 8 total, 8 up, 7 in 2024-03-26T21:43:28.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:43:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:43:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/62763886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[40587]: osdmap e329: 8 total, 8 up, 7 in 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/62763886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T21:43:28.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-mon[29406]: osdmap e329: 8 total, 8 up, 7 in 2024-03-26T21:43:28.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:28.645 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:28.733 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:28.735 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:28.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:28.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-26T21:43:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:28.782 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:28.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:28.903 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:28.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:29.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:29.032 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:29.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.158 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:29.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:29.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:29.263 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:29 smithi055 ceph-mon[40587]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:43:29.263 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:29 smithi055 ceph-mon[40587]: osdmap e330: 8 total, 8 up, 7 in 2024-03-26T21:43:29.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:29 smithi055 ceph-mon[29406]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:43:29.264 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:29 smithi055 ceph-mon[29406]: osdmap e330: 8 total, 8 up, 7 in 2024-03-26T21:43:29.264 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:29.287 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:29.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:29.417 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:29 smithi067 ceph-mon[33346]: pgmap v675: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:43:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:29 smithi067 ceph-mon[33346]: osdmap e330: 8 total, 8 up, 7 in 2024-03-26T21:43:29.519 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:29.542 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:29.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:29.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:29.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:29.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:29.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:29.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:30.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:30.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:30.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:30.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:30.301 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:30.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:30.427 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:30.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:30.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:30.683 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:30.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:30.808 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: pgmap v678: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: pgmap v678: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:30.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:30 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:30.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:30.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: pgmap v678: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:43:30.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:30 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:31.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:31.165 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.190 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:31.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:31.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:31.316 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:31.320 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:31.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:31.445 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:31.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:31.575 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:31.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:31.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:31.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.830 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:31.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:31.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:31.954 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:31.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:32.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:32.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:32.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:32.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:32.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:32.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:32.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:32.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:32.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:32.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:32.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:32.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:32.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:32.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:32.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[40587]: pgmap v680: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:43:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[40587]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T21:43:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[29406]: pgmap v680: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:43:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[29406]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T21:43:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:33.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:33.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:33.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:33.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:32 smithi067 ceph-mon[33346]: pgmap v680: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:43:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:32 smithi067 ceph-mon[33346]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T21:43:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:33.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:33.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:33.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:33.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:33.602 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:33.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:33.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:33.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:33.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147831 9.8 9:18de0198:::benchmark_data_smithi055_135863_object17180:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 148.26393322199999, 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0240032659999998, 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.925 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147831 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.926 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666343+0000", 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4470000000000001e-06 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666347+0000", 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6449999999999998e-06 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666403+0000", 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6029000000000002e-05 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666414+0000", 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1065e-05 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.675183+0000", 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087694560000000001 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.675195+0000", 2024-03-26T21:43:33.929 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1766999999999999e-05 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661090+0000", 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98589458299999999 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661176+0000", 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6285999999999993e-05 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.930 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.688687+0000", 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0275108209999999 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.690345+0000", 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016581670000000001 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.931 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147849 9.8 9:137978a5:::benchmark_data_smithi055_135863_object17198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 148.238520851, 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.011147448, 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147849 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.932 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691756+0000", 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.652e-06 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691764+0000", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1960000000000008e-06 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691784+0000", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9714000000000001e-05 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691795+0000", 2024-03-26T21:43:33.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1756999999999999e-05 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692479+0000", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00068378600000000003 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692492+0000", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2384e-05 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661266+0000", 2024-03-26T21:43:33.935 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.968774358 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661276+0000", 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9159999999999996e-06 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.691612+0000", 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0303359299999999 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.936 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.692227+0000", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00061501800000000005 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.692352+0000", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012515599999999999 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702314+0000", 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0099620539999999997 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702361+0000", 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6623999999999999e-05 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702557+0000", 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019608900000000001 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.938 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702766+0000", 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000209464 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702902+0000", 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013535000000000001 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.939 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.940 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.940 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.940 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.940 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147857 9.8 9:1c81c588:::benchmark_data_smithi055_135863_object17206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 148.21448781800001, 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.990900015, 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.941 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147857 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.942 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715789+0000", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.759e-06 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715796+0000", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.883e-06 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715810+0000", 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4405e-05 2024-03-26T21:43:33.943 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715820+0000", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.781e-06 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715895+0000", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4951999999999997e-05 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715906+0000", 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.063e-05 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661303+0000", 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94539728300000003 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661311+0000", 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5790000000000004e-06 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.695645+0000", 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0343331120000001 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.695785+0000", 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014065 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.696089+0000", 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030426299999999998 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702539+0000", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0064498619999999998 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702910+0000", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00037042599999999998 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706506+0000", 2024-03-26T21:43:33.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0035965770000000001 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706567+0000", 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0748000000000002e-05 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706687+0000", 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120105 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166764 9.8 9:14fe2036:::benchmark_data_smithi055_135863_object36113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 129.345004155, 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.166361408, 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.949 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166764 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.950 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585272+0000", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4699999999999999e-06 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585278+0000", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7960000000000001e-06 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585291+0000", 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3303e-05 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585300+0000", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4849999999999992e-06 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585900+0000", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00059990599999999999 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585980+0000", 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0327000000000005e-05 2024-03-26T21:43:33.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.586074+0000", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4301000000000004e-05 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.590001+0000", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0039261749999999996 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.604749+0000", 2024-03-26T21:43:33.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.014748618999999999 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.604764+0000", 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4477e-05 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751549+0000", 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.146784848 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751632+0000", 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3701000000000004e-05 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166778 9.8 9:1f03b6ad:::benchmark_data_smithi055_135863_object36127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 129.31025108399999, 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.139688762, 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.955 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166778 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.956 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620025+0000", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.53e-06 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620033+0000", 2024-03-26T21:43:33.957 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1550000000000001e-06 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620047+0000", 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4883e-05 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620057+0000", 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6250000000000002e-06 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.958 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620136+0000", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8548999999999996e-05 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620159+0000", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3354e-05 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751880+0000", 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1317214899999999 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.959 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752004+0000", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012312299999999999 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752153+0000", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014930600000000001 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758277+0000", 2024-03-26T21:43:33.960 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0061243060000000004 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758655+0000", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00037819600000000002 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758697+0000", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2063e-05 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759530+0000", 2024-03-26T21:43:33.961 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00083214400000000005 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759595+0000", 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4970999999999998e-05 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759713+0000", 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011806700000000001 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.962 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166798 9.5 9:a9aae264:::benchmark_data_smithi055_135863_object36147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 128.175139777, 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99086638599999999, 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166798 2024-03-26T21:43:33.963 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755137+0000", 2024-03-26T21:43:33.964 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.635e-06 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755142+0000", 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.921e-06 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755167+0000", 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4850000000000001e-05 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.965 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755177+0000", 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0069e-05 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755250+0000", 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3651000000000003e-05 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.966 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755263+0000", 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2627e-05 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733027+0000", 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97776415900000002 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733159+0000", 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013211200000000001 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.967 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733292+0000", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013281999999999999 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.738986+0000", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056941980000000001 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742091+0000", 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003104711 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.968 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.745825+0000", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037338330000000002 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.745889+0000", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4539e-05 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746002+0000", 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112261 2024-03-26T21:43:33.969 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166806 9.5 9:ab01a6aa:::benchmark_data_smithi055_135863_object36155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 128.16056240200001, 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97653648699999995, 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.970 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166806 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.971 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769714+0000", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3990000000000001e-06 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769721+0000", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5719999999999999e-06 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769734+0000", 2024-03-26T21:43:33.972 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3638000000000001e-05 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769743+0000", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4149999999999999e-06 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769817+0000", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4548999999999994e-05 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769829+0000", 2024-03-26T21:43:33.973 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.166e-05 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736190+0000", 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96636086799999998 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736300+0000", 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011020599999999999 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:33.974 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736415+0000", 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115605 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742108+0000", 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056924000000000002 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.975 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742327+0000", 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021956800000000001 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746057+0000", 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037296909999999998 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746114+0000", 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6436e-05 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746249+0000", 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013548 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241352 9.8 9:1a77692d:::benchmark_data_smithi055_135863_object110701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:43:33.977 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 44.053824980000002, 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96528207899999996, 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241352 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.978 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876452+0000", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1229999999999998e-06 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876461+0000", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9670000000000007e-06 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876481+0000", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0361e-05 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876493+0000", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.169e-05 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876566+0000", 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3020000000000002e-05 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876576+0000", 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9399999999999997e-06 2024-03-26T21:43:33.981 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836333+0000", 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.95975673500000003 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836431+0000", 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8459e-05 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836511+0000", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0210999999999995e-05 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841566+0000", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050547559999999997 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841607+0000", 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1267999999999999e-05 2024-03-26T21:43:33.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841648+0000", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0703e-05 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841697+0000", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9001000000000001e-05 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841732+0000", 2024-03-26T21:43:33.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4845000000000003e-05 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241370 9.8 9:1a2077f5:::benchmark_data_smithi055_135863_object110719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 44.031562872000002, 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94483976700000005, 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.985 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241370 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:43:33.986 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898714+0000", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.621e-06 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898721+0000", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3529999999999997e-06 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898737+0000", 2024-03-26T21:43:33.987 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6283e-05 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898747+0000", 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2979999999999997e-06 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898812+0000", 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5160000000000006e-05 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898823+0000", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0910000000000001e-05 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838884+0000", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94006149800000005 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838956+0000", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2070999999999995e-05 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:33.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.839026+0000", 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9647999999999995e-05 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841779+0000", 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0027533039999999998 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843421+0000", 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016415379999999999 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843462+0000", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0815999999999997e-05 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843511+0000", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9166999999999999e-05 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843552+0000", 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1100000000000003e-05 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:33.991 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241386 9.8 9:15e63956:::benchmark_data_smithi055_135863_object110735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 44.015286682000003, 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92876690900000003, 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241386 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:33.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:33.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914989+0000", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1060000000000001e-06 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914994+0000", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.04e-06 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915006+0000", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1435e-05 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:33.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915014+0000", 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7619999999999995e-06 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915071+0000", 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7256000000000002e-05 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915079+0000", 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3780000000000001e-06 2024-03-26T21:43:33.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840207+0000", 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92512746099999998 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840287+0000", 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0653000000000002e-05 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840368+0000", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0980000000000001e-05 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843430+0000", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030614000000000001 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843613+0000", 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000182805 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:33.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:33.998 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:33.998 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843672+0000", 2024-03-26T21:43:33.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9079000000000001e-05 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843722+0000", 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0284000000000002e-05 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843755+0000", 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3269999999999998e-05 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241396 9.8 9:15ab1784:::benchmark_data_smithi055_135863_object110745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 44.008876485999998, 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.928642615, 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.001 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241396 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921399+0000", 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.032e-06 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921405+0000", 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0209999999999999e-06 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.003 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921417+0000", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2874000000000001e-05 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921426+0000", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1429999999999998e-06 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921482+0000", 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6236999999999997e-05 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921490+0000", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9610000000000005e-06 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841812+0000", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92032255900000004 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841909+0000", 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7065999999999995e-05 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.842006+0000", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6650999999999993e-05 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845746+0000", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003739839 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845805+0000", 2024-03-26T21:43:34.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9500000000000003e-05 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.849965+0000", 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0041593410000000004 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850001+0000", 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6341000000000003e-05 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850041+0000", 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0049999999999998e-05 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241442 9.8 9:117f2a93:::benchmark_data_smithi055_135863_object110791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.972568641999999, 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90324378800000005, 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.008 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241442 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957708+0000", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.956e-06 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957726+0000", 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8070000000000001e-05 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957742+0000", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5252000000000001e-05 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957750+0000", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1359999999999997e-06 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957831+0000", 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1278000000000003e-05 2024-03-26T21:43:34.011 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957843+0000", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2038999999999999e-05 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843789+0000", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88594599600000001 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843888+0000", 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8515000000000001e-05 2024-03-26T21:43:34.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843989+0000", 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000101331 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852756+0000", 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087672730000000008 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852772+0000", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5968999999999999e-05 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860849+0000", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0080765820000000006 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860901+0000", 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2301e-05 2024-03-26T21:43:34.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860950+0000", 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9089999999999999e-05 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241445 9.8 9:195f423d:::benchmark_data_smithi055_135863_object110794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.971574859999997, 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.902364579, 2024-03-26T21:43:34.015 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241445 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958701+0000", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4399999999999999e-07 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958706+0000", 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.091e-06 2024-03-26T21:43:34.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958721+0000", 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4491999999999999e-05 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958729+0000", 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.541e-06 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958812+0000", 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2684999999999999e-05 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958822+0000", 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0095000000000001e-05 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845839+0000", 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88701723600000004 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845945+0000", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105958 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.846050+0000", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104899 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852779+0000", 2024-03-26T21:43:34.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0067287370000000003 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852812+0000", 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3673000000000001e-05 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860989+0000", 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081766100000000008 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861031+0000", 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1711000000000001e-05 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861065+0000", 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4007000000000002e-05 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241482 9.8 9:1973c4a0:::benchmark_data_smithi055_135863_object110831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:43:34.022 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.942737860999998, 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88112820999999997, 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241482 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987539+0000", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5999999999999999e-06 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987545+0000", 2024-03-26T21:43:34.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2820000000000002e-06 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987558+0000", 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2744e-05 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987566+0000", 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1259999999999998e-06 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987642+0000", 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6149999999999994e-05 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987652+0000", 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0351e-05 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847815+0000", 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.86016230900000001 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847915+0000", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000100707 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847989+0000", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3889999999999999e-05 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860773+0000", 2024-03-26T21:43:34.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012783341 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861101+0000", 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032820899999999999 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868358+0000", 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072569499999999999 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868618+0000", 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025993400000000002 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868665+0000", 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7617000000000002e-05 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241559 9.8 9:12082b59:::benchmark_data_smithi055_135863_object110908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.049928+0000", 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.880346664000001, 2024-03-26T21:43:34.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.81887654300000001, 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241559 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049928+0000", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049928+0000", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049930+0000", 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.407e-06 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049936+0000", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.37e-06 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049947+0000", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1365e-05 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.049956+0000", 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2919999999999996e-06 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.032 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.050020+0000", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4120000000000003e-05 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.050028+0000", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8779999999999998e-06 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850094+0000", 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.800066728 2024-03-26T21:43:34.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850276+0000", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018119300000000001 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850422+0000", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014589299999999999 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860812+0000", 2024-03-26T21:43:34.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010390450000000001 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861139+0000", 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032664400000000001 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868726+0000", 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.007587468 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868767+0000", 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0664999999999998e-05 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868805+0000", 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.807e-05 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241589 9.8 9:1d24a686:::benchmark_data_smithi055_135863_object110938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.090959+0000", 2024-03-26T21:43:34.036 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.839315814000003, 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.77792652399999995, 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241589 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090959+0000", 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.037 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090959+0000", 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090960+0000", 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4e-07 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.038 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090964+0000", 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0720000000000001e-06 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090971+0000", 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6680000000000004e-06 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.090976+0000", 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.541e-06 2024-03-26T21:43:34.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.091031+0000", 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4289999999999997e-05 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.091039+0000", 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1850000000000004e-06 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852847+0000", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.76180876600000003 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.853007+0000", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015908799999999999 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.853271+0000", 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00026402499999999998 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868222+0000", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.014951742000000001 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868340+0000", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117316 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868812+0000", 2024-03-26T21:43:34.042 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00047215000000000001 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868850+0000", 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7710999999999999e-05 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868886+0000", 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6130000000000001e-05 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241628 9.8 9:1e7ebf5c:::benchmark_data_smithi055_135863_object110977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.133600+0000", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.796674600000003, 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.73536584999999999, 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241628 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.044 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133600+0000", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133600+0000", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133602+0000", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.37e-06 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133607+0000", 2024-03-26T21:43:34.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4530000000000001e-06 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133618+0000", 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1204e-05 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133627+0000", 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9800000000000004e-06 2024-03-26T21:43:34.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133701+0000", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3552000000000003e-05 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.133711+0000", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8230000000000006e-06 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.855941+0000", 2024-03-26T21:43:34.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.72223011500000001 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.856065+0000", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012406 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.856247+0000", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000181867 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868269+0000", 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012022645 2024-03-26T21:43:34.048 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868714+0000", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00044442400000000001 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868892+0000", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017800199999999999 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868931+0000", 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9214000000000001e-05 2024-03-26T21:43:34.049 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868966+0000", 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5141000000000001e-05 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241630 9.8 9:140135d2:::benchmark_data_smithi055_135863_object110979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.135371+0000", 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.794903693999998, 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.74090937700000004, 2024-03-26T21:43:34.050 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241630 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135371+0000", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.051 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135371+0000", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135372+0000", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.173e-06 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135377+0000", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5669999999999996e-06 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135387+0000", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0251999999999999e-05 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135394+0000", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.007e-06 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135438+0000", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4115e-05 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.135444+0000", 2024-03-26T21:43:34.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7069999999999997e-06 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861179+0000", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.725734608 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861299+0000", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120537 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861422+0000", 2024-03-26T21:43:34.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000122652 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868973+0000", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0075510129999999996 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869010+0000", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6801999999999998e-05 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876112+0000", 2024-03-26T21:43:34.055 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0071020800000000002 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876229+0000", 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117055 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876281+0000", 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1808999999999998e-05 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.056 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241651 9.8 9:1077545c:::benchmark_data_smithi055_135863_object111000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.154654+0000", 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.775621059999999, 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.72213657499999995, 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.057 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241651 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154654+0000", 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154654+0000", 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154655+0000", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0419999999999999e-06 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154660+0000", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2050000000000001e-06 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154672+0000", 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.164e-05 2024-03-26T21:43:34.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154679+0000", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6240000000000002e-06 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154737+0000", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7386000000000002e-05 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.154746+0000", 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4089999999999992e-06 2024-03-26T21:43:34.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.864271+0000", 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.70952483300000002 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.864378+0000", 2024-03-26T21:43:34.061 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010715199999999999 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.864816+0000", 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000437331 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.062 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869056+0000", 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0042403400000000004 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869068+0000", 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2214e-05 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876332+0000", 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072635740000000001 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876748+0000", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00041663399999999998 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876790+0000", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2191000000000003e-05 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241663 9.8 9:142db58f:::benchmark_data_smithi055_135863_object111012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:50.166436+0000", 2024-03-26T21:43:34.064 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 43.763838808000003, 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.713901495, 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241663 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166436+0000", 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.065 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166436+0000", 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166438+0000", 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4160000000000001e-06 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166443+0000", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.446e-06 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166455+0000", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1567e-05 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166463+0000", 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9519999999999994e-06 2024-03-26T21:43:34.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166537+0000", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4685999999999996e-05 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.166547+0000", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0200999999999999e-05 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869094+0000", 2024-03-26T21:43:34.068 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.70254665599999999 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869384+0000", 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029011199999999998 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.869523+0000", 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000139312 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:43:34.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.876294+0000", 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.006770514 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.880199+0000", 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003904509 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.880240+0000", 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1069000000000003e-05 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.880290+0000", 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0016999999999999e-05 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.880338+0000", 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8038000000000003e-05 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:43:34.071 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:43:34.072 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:33 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:43:34.072 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:33 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:43:34.072 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:33 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:43:34.072 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:33 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:43:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:34.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:34.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:34.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:33 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:43:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:33 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:43:34.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:34.429 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:34.453 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:34.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:34.577 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:34.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:34.703 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:34.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:34.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:34.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:34.953 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:34.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:35.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:35.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:34 smithi055 ceph-mon[29406]: pgmap v682: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:43:35.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:34 smithi055 ceph-mon[29406]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T21:43:35.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:34 smithi055 ceph-mon[40587]: pgmap v682: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:43:35.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:34 smithi055 ceph-mon[40587]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T21:43:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:35.183 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:35.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:34 smithi067 ceph-mon[33346]: pgmap v682: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:43:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:34 smithi067 ceph-mon[33346]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T21:43:35.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:35.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:35.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:35.462 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:35.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:35.590 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:35.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:35.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:35.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:35.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:35.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:35.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:36.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:43:36.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.123 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:36.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:36.223 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:36.248 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:36.327 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:43:36.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:43:36.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:36.366 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:43:36.377 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:36.387 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:43:36.407 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:43:36.427 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:43:36.447 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:43:36.467 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:43:36.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:43:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:43:36.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:36.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.631 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:36.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:36.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:36.757 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:36.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:36.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:36.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:36.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:37.012 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[40587]: pgmap v684: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3171213677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[40587]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T21:43:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[29406]: pgmap v684: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3171213677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:36 smithi055 ceph-mon[29406]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T21:43:37.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:37.140 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:36 smithi067 ceph-mon[33346]: pgmap v684: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3171213677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:36 smithi067 ceph-mon[33346]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T21:43:37.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:37.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:37.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:37.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:37.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:37.651 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:37.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:37.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:37.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:37.908 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:37.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:38.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:38.017 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:38.018 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:38.036 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:38.056 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:38.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:38.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:38.290 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:38.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:38.416 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:38.518 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:38.542 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:38.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:38.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:38.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:38.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:38.817 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-26T21:43:38.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:38.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:38.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:39.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:39.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:38 smithi055 ceph-mon[29406]: pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:39.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:43:39.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:39.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:38 smithi055 ceph-mon[40587]: pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:39.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:43:39.139 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:38 smithi067 ceph-mon[33346]: pgmap v686: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:43:39.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:39.178 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:39.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:39.302 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:39.404 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:39.427 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:39.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:39.552 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:39.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:39.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:43:39.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:39.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:39.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:39.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:40.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:40.060 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:40.098 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:39 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:43:40.098 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:39 smithi055 ceph-mon[40587]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T21:43:40.098 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:39 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:43:40.098 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:39 smithi055 ceph-mon[29406]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T21:43:40.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:40.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:39 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3051396302' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:43:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:39 smithi067 ceph-mon[33346]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T21:43:40.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:40.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:40.442 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:40.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:40.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:40.698 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:40.723 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:43:40.724 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:43:40.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:40.824 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:40.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:40.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:40.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:40.974 DEBUG:teuthology.orchestra.run.smithi055:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-26T21:43:40.974 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-26T21:43:40.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:43:40.993 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T21:43:40.993 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T21:43:41.015 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[40587]: pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:41.015 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:43:41.015 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:43:41.015 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[40587]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[29406]: pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:43:41.016 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:40 smithi055 ceph-mon[29406]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T21:43:41.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:41.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:41.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:40 smithi067 ceph-mon[33346]: pgmap v688: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:40 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:43:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:40 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:43:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:40 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:43:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:40 smithi067 ceph-mon[33346]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T21:43:41.239 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:41.252 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:hints = 1 2024-03-26T21:43:41.252 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T21:43:41.252 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object prefix: benchmark_data_smithi055_163313 2024-03-26T21:43:41.252 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:43:41.252 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 0 0 0 0 0 0 - 0 2024-03-26T21:43:41.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:41.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:41.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:41.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:41.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:41.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:41.767 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:41.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:43:41.863 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:43:41.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:41.948 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:41.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:42.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:42.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3152191013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:41 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3152191013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:41 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:42.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:42.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3152191013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:41 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:43:42.254 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 1 16 3577 3561 222.243 222.562 0.00414618 0.00448608 2024-03-26T21:43:42.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:42.370 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:42.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:42.511 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:42.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:42.752 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:42.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:42.802 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:42.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:42.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:42.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:43.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:43.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:43.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:43:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:42 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:43:43.254 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 2 15 7037 7022 219.271 216.312 0.00351555 0.00454958 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: pgmap v690: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:43:43.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:43:43.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:43:43.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T21:43:43.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:43:43.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:42 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:43:43.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:43.343 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:43.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:43.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:43.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:43.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:43.868 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:43.905 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:43.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:44.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[40587]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[29406]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:43:44.155 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:43 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:43:44.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:44.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:43 smithi067 ceph-mon[33346]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T21:43:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:43 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:43:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:43 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:43:44.254 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 3 16 10303 10287 214.172 204.062 0.00485393 0.00466234 2024-03-26T21:43:44.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:44.339 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:44.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:44.478 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:44.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:44.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:44.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:44.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:44.896 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:44.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:44.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:44.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[40587]: pgmap v693: 148 pgs: 148 active+clean; 135 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 28 MiB/s wr, 448 op/s 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[29406]: pgmap v693: 148 pgs: 148 active+clean; 135 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 28 MiB/s wr, 448 op/s 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:43:45.000 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:44 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:43:45.036 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:45.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:45.174 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:44 smithi067 ceph-mon[33346]: pgmap v693: 148 pgs: 148 active+clean; 135 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 28 MiB/s wr, 448 op/s 2024-03-26T21:43:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:44 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:43:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:44 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:43:45.254 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 4 16 13241 13225 206.535 183.625 0.00501391 0.00483555 2024-03-26T21:43:45.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:45.315 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:45.350 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:43:45.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:45.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:45.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:45.595 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:45.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:45.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:45.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:45.873 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:45.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:45.946 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:43:45.946 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[40587]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[29406]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:43:45.947 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:45 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:43:45.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:46.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:46.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:45 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:43:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:45 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:43:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:45 smithi067 ceph-mon[33346]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T21:43:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:45 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:43:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:45 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:43:46.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:46.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 5 15 15992 15977 199.603 172 0.00819162 0.0050024 2024-03-26T21:43:46.289 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:46.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:46.528 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:46.566 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:46.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:46.705 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:46.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:46.860 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:46.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: pgmap v695: 148 pgs: 148 active+clean; 260 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 885 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:43:46.956 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[40587]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: pgmap v695: 148 pgs: 148 active+clean; 260 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 885 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:43:46.957 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:46 smithi055 ceph-mon[29406]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T21:43:46.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:47.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:47.047 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:43:47.084 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:43:47.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:47.124 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:43:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:47.165 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:43:47.205 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: pgmap v695: 148 pgs: 148 active+clean; 260 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 885 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:43:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:46 smithi067 ceph-mon[33346]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T21:43:47.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:47.249 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:43:47.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 6 16 18685 18669 194.377 168.25 0.00672092 0.00513798 2024-03-26T21:43:47.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:47.296 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:43:47.335 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:43:47.342 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:43:47.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:47.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:47.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:47.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:47.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:47.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:47.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:47.830 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:47.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:47.966 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:47.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:48.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:48.106 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/307970583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/307970583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:48.200 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:47 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:48.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:47 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/307970583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:43:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:47 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:43:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:43:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:47 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:48.251 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:48.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 7 15 21215 21200 189.207 158.188 0.00834448 0.00527862 2024-03-26T21:43:48.309 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:43:48.309 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:43:48.310 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:43:48.310 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:43:48.310 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:43:48.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:48.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:48.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:48.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:48.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:48.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:48.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:48.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:48.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:49.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 8 16 23793 23777 185.688 161.062 0.00440004 0.00537997 2024-03-26T21:43:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: pgmap v697: 148 pgs: 148 active+clean; 862 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 889 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: pgmap v697: 148 pgs: 148 active+clean; 862 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 889 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:43:49.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:43:49.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:49 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:43:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:49.367 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:49.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:49.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: pgmap v697: 148 pgs: 148 active+clean; 862 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 889 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:43:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:49 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:43:49.508 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:49.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:49.610 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:49.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:49.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:49.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:49.782 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:49.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:49.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:49.921 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:49.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:50.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:50.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:50.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 9 16 26058 26042 180.785 141.562 0.0065555 0.00552647 2024-03-26T21:43:50.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:50.343 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:50.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:50.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:50 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:43:50.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:50 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:43:50.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:50 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:43:50.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:50 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:43:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:43:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:43:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:43:50.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:43:50.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:43:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:43:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:43:50.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:50 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:43:50.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:50.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:50.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:50.748 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:50.849 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:43:50.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:50.853 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:43:50.889 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:50.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:50.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:51.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:51.255 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 10 16 27943 27927 174.486 117.812 0.0132057 0.0057259 2024-03-26T21:43:51.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: pgmap v698: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:43:51.360 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: pgmap v698: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:43:51.361 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:51 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:43:51.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:51.445 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: pgmap v698: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:43:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:51 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:43:51.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:51.583 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:51.684 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:51.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:51.822 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:51.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:51.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:51.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:51.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:52.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:52.126 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:52.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:52.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 11 16 30173 30157 171.294 139.375 0.00599322 0.00583246 2024-03-26T21:43:52.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:52.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:52.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:52 smithi067 ceph-mon[33346]: from='client.25484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:52 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:43:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:52 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:43:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:52 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:43:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:52 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:43:52.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:52.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[29406]: from='client.25484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:52.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:43:52.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:43:52.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[40587]: from='client.25484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:43:52.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:52 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:43:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:52.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:52.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:52.826 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:52.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:52.963 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:52.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:53.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:53.103 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:53.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:53.243 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:53.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 12 16 32205 32189 167.603 127 0.00286955 0.00596106 2024-03-26T21:43:53.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:53.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:53.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:53 smithi067 ceph-mon[33346]: pgmap v699: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T21:43:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:53 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:43:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:53 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:43:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:53 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:43:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:53 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:43:53.520 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[29406]: pgmap v699: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T21:43:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:43:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:43:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:43:53.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:43:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[40587]: pgmap v699: 148 pgs: 148 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T21:43:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:43:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:43:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:43:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:53 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:43:53.622 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:53.659 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2024-03-26T21:43:49.620493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1252071848,"num_objects":19294,"num_object_clones":0,"num_object_copies":57882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19294,"num_whiteouts":0,"num_read":2111,"num_read_kb":1959,"num_write":19997,"num_write_kb":1224605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1775891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20097,"ondisk_log_size":20097,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6884696,"kb_used_data":3679116,"kb_used_omap":18,"kb_used_meta":3205485,"kb_avail":649331368,"statfs":{"total":671965249536,"available":664915320832,"internally_reserved":0,"allocated":3767414784,"data_stored":3762909609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19291,"internal_metadata":3282416805},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1139408896,"num_objects":17386,"num_object_clones":0,"num_object_copies":52158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17386,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17386,"num_write_kb":1112704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.759260"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":307,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802400+0000","last_change":"2024-03-26T21:41:48.911741+0000","last_active":"2024-03-26T21:43:46.802400+0000","last_peered":"2024-03-26T21:43:46.802400+0000","last_clean":"2024-03-26T21:43:46.802400+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:43:46.802400+0000","last_undegraded":"2024-03-26T21:43:46.802400+0000","last_fullsized":"2024-03-26T21:43:46.802400+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:48.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:41:48.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801629+0000","last_change":"2024-03-26T21:43:46.178825+0000","last_active":"2024-03-26T21:43:46.801629+0000","last_peered":"2024-03-26T21:43:46.801629+0000","last_clean":"2024-03-26T21:43:46.801629+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:43:46.801629+0000","last_undegraded":"2024-03-26T21:43:46.801629+0000","last_fullsized":"2024-03-26T21:43:46.801629+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:46.178692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:46.178692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:12.963816+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":604,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806060+0000","last_change":"2024-03-26T21:40:20.651612+0000","last_active":"2024-03-26T21:43:46.806060+0000","last_peered":"2024-03-26T21:43:46.806060+0000","last_clean":"2024-03-26T21:43:46.806060+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:43:46.806060+0000","last_undegraded":"2024-03-26T21:43:46.806060+0000","last_fullsized":"2024-03-26T21:43:46.806060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.651494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.651494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":334,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801763+0000","last_change":"2024-03-26T21:43:16.802497+0000","last_active":"2024-03-26T21:43:46.801763+0000","last_peered":"2024-03-26T21:43:46.801763+0000","last_clean":"2024-03-26T21:43:46.801763+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:43:46.801763+0000","last_undegraded":"2024-03-26T21:43:46.801763+0000","last_fullsized":"2024-03-26T21:43:46.801763+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:40:26.943893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:26.943893+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":138,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807626+0000","last_change":"2024-03-26T21:41:26.600406+0000","last_active":"2024-03-26T21:43:46.807626+0000","last_peered":"2024-03-26T21:43:46.807626+0000","last_clean":"2024-03-26T21:43:46.807626+0000","last_became_active":"2024-03-26T21:41:26.600082+0000","last_became_peered":"2024-03-26T21:41:26.600082+0000","last_unstale":"2024-03-26T21:43:46.807626+0000","last_undegraded":"2024-03-26T21:43:46.807626+0000","last_fullsized":"2024-03-26T21:43:46.807626+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:20.336684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:41:20.336684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":307,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801914+0000","last_change":"2024-03-26T21:41:38.083671+0000","last_active":"2024-03-26T21:43:46.801914+0000","last_peered":"2024-03-26T21:43:46.801914+0000","last_clean":"2024-03-26T21:43:46.801914+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:43:46.801914+0000","last_undegraded":"2024-03-26T21:43:46.801914+0000","last_fullsized":"2024-03-26T21:43:46.801914+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.083526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":159,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.882250+0000","last_change":"2024-03-26T21:42:57.090990+0000","last_active":"2024-03-26T21:43:44.882250+0000","last_peered":"2024-03-26T21:43:44.882250+0000","last_clean":"2024-03-26T21:43:44.882250+0000","last_became_active":"2024-03-26T21:41:06.689493+0000","last_became_peered":"2024-03-26T21:41:06.689493+0000","last_unstale":"2024-03-26T21:43:44.882250+0000","last_undegraded":"2024-03-26T21:43:44.882250+0000","last_fullsized":"2024-03-26T21:43:44.882250+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:57.090868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.090868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:57.877867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":587,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809560+0000","last_change":"2024-03-26T21:42:43.232904+0000","last_active":"2024-03-26T21:43:46.809560+0000","last_peered":"2024-03-26T21:43:46.809560+0000","last_clean":"2024-03-26T21:43:46.809560+0000","last_became_active":"2024-03-26T21:41:06.687399+0000","last_became_peered":"2024-03-26T21:41:06.687399+0000","last_unstale":"2024-03-26T21:43:46.809560+0000","last_undegraded":"2024-03-26T21:43:46.809560+0000","last_fullsized":"2024-03-26T21:43:46.809560+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:43.232749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:43.232749+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.944023+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":592,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807017+0000","last_change":"2024-03-26T21:41:25.610223+0000","last_active":"2024-03-26T21:43:46.807017+0000","last_peered":"2024-03-26T21:43:46.807017+0000","last_clean":"2024-03-26T21:43:46.807017+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:43:46.807017+0000","last_undegraded":"2024-03-26T21:43:46.807017+0000","last_fullsized":"2024-03-26T21:43:46.807017+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.492302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.492302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":611,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809491+0000","last_change":"2024-03-26T21:43:44.320137+0000","last_active":"2024-03-26T21:43:46.809491+0000","last_peered":"2024-03-26T21:43:46.809491+0000","last_clean":"2024-03-26T21:43:46.809491+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:43:46.809491+0000","last_undegraded":"2024-03-26T21:43:46.809491+0000","last_fullsized":"2024-03-26T21:43:46.809491+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:44.319965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:43:44.319965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:45.582445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":623,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809898+0000","last_change":"2024-03-26T21:40:42.938835+0000","last_active":"2024-03-26T21:43:46.809898+0000","last_peered":"2024-03-26T21:43:46.809898+0000","last_clean":"2024-03-26T21:43:46.809898+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:43:46.809898+0000","last_undegraded":"2024-03-26T21:43:46.809898+0000","last_fullsized":"2024-03-26T21:43:46.809898+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:42.938743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:40:42.938743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":342,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802509+0000","last_change":"2024-03-26T21:41:53.752959+0000","last_active":"2024-03-26T21:43:46.802509+0000","last_peered":"2024-03-26T21:43:46.802509+0000","last_clean":"2024-03-26T21:43:46.802509+0000","last_became_active":"2024-03-26T21:41:06.682256+0000","last_became_peered":"2024-03-26T21:41:06.682256+0000","last_unstale":"2024-03-26T21:43:46.802509+0000","last_undegraded":"2024-03-26T21:43:46.802509+0000","last_fullsized":"2024-03-26T21:43:46.802509+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:53.752832+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:41:53.752832+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":604,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809805+0000","last_change":"2024-03-26T21:41:06.691334+0000","last_active":"2024-03-26T21:43:46.809805+0000","last_peered":"2024-03-26T21:43:46.809805+0000","last_clean":"2024-03-26T21:43:46.809805+0000","last_became_active":"2024-03-26T21:41:06.690326+0000","last_became_peered":"2024-03-26T21:41:06.690326+0000","last_unstale":"2024-03-26T21:43:46.809805+0000","last_undegraded":"2024-03-26T21:43:46.809805+0000","last_fullsized":"2024-03-26T21:43:46.809805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:48.999826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.999826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":270,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807172+0000","last_change":"2024-03-26T21:41:25.658468+0000","last_active":"2024-03-26T21:43:46.807172+0000","last_peered":"2024-03-26T21:43:46.807172+0000","last_clean":"2024-03-26T21:43:46.807172+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:43:46.807172+0000","last_undegraded":"2024-03-26T21:43:46.807172+0000","last_fullsized":"2024-03-26T21:43:46.807172+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:28.958626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:28.958626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":623,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809837+0000","last_change":"2024-03-26T21:40:35.930617+0000","last_active":"2024-03-26T21:43:46.809837+0000","last_peered":"2024-03-26T21:43:46.809837+0000","last_clean":"2024-03-26T21:43:46.809837+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:43:46.809837+0000","last_undegraded":"2024-03-26T21:43:46.809837+0000","last_fullsized":"2024-03-26T21:43:46.809837+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_clean_scrub_stamp":"2024-03-26T21:40:35.930502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":328,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802232+0000","last_change":"2024-03-26T21:43:02.671358+0000","last_active":"2024-03-26T21:43:46.802232+0000","last_peered":"2024-03-26T21:43:46.802232+0000","last_clean":"2024-03-26T21:43:46.802232+0000","last_became_active":"2024-03-26T21:41:06.682153+0000","last_became_peered":"2024-03-26T21:41:06.682153+0000","last_unstale":"2024-03-26T21:43:46.802232+0000","last_undegraded":"2024-03-26T21:43:46.802232+0000","last_fullsized":"2024-03-26T21:43:46.802232+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:43:02.671224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:02.671224+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.142642+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":300,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803386+0000","last_change":"2024-03-26T21:40:40.874687+0000","last_active":"2024-03-26T21:43:46.803386+0000","last_peered":"2024-03-26T21:43:46.803386+0000","last_clean":"2024-03-26T21:43:46.803386+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:43:46.803386+0000","last_undegraded":"2024-03-26T21:43:46.803386+0000","last_fullsized":"2024-03-26T21:43:46.803386+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.874570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.874570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803108+0000","last_change":"2024-03-26T21:40:16.881228+0000","last_active":"2024-03-26T21:43:46.803108+0000","last_peered":"2024-03-26T21:43:46.803108+0000","last_clean":"2024-03-26T21:43:46.803108+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:43:46.803108+0000","last_undegraded":"2024-03-26T21:43:46.803108+0000","last_fullsized":"2024-03-26T21:43:46.803108+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.881096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":625,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805321+0000","last_change":"2024-03-26T21:42:56.212250+0000","last_active":"2024-03-26T21:43:46.805321+0000","last_peered":"2024-03-26T21:43:46.805321+0000","last_clean":"2024-03-26T21:43:46.805321+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:43:46.805321+0000","last_undegraded":"2024-03-26T21:43:46.805321+0000","last_fullsized":"2024-03-26T21:43:46.805321+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.212112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.563850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":688,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809763+0000","last_change":"2024-03-26T21:42:57.266611+0000","last_active":"2024-03-26T21:43:46.809763+0000","last_peered":"2024-03-26T21:43:46.809763+0000","last_clean":"2024-03-26T21:43:46.809763+0000","last_became_active":"2024-03-26T21:41:06.694407+0000","last_became_peered":"2024-03-26T21:41:06.694407+0000","last_unstale":"2024-03-26T21:43:46.809763+0000","last_undegraded":"2024-03-26T21:43:46.809763+0000","last_fullsized":"2024-03-26T21:43:46.809763+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:57.266450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.266450+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.961671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":168,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.883420+0000","last_change":"2024-03-26T21:42:59.247357+0000","last_active":"2024-03-26T21:43:44.883420+0000","last_peered":"2024-03-26T21:43:44.883420+0000","last_clean":"2024-03-26T21:43:44.883420+0000","last_became_active":"2024-03-26T21:41:06.689834+0000","last_became_peered":"2024-03-26T21:41:06.689834+0000","last_unstale":"2024-03-26T21:43:44.883420+0000","last_undegraded":"2024-03-26T21:43:44.883420+0000","last_fullsized":"2024-03-26T21:43:44.883420+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.247230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.247230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.613979+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":317,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803125+0000","last_change":"2024-03-26T21:41:47.917327+0000","last_active":"2024-03-26T21:43:46.803125+0000","last_peered":"2024-03-26T21:43:46.803125+0000","last_clean":"2024-03-26T21:43:46.803125+0000","last_became_active":"2024-03-26T21:41:26.595687+0000","last_became_peered":"2024-03-26T21:41:26.595687+0000","last_unstale":"2024-03-26T21:43:46.803125+0000","last_undegraded":"2024-03-26T21:43:46.803125+0000","last_fullsized":"2024-03-26T21:43:46.803125+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:47.917202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:41:47.917202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":299,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802393+0000","last_change":"2024-03-26T21:40:29.902688+0000","last_active":"2024-03-26T21:43:46.802393+0000","last_peered":"2024-03-26T21:43:46.802393+0000","last_clean":"2024-03-26T21:43:46.802393+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:43:46.802393+0000","last_undegraded":"2024-03-26T21:43:46.802393+0000","last_fullsized":"2024-03-26T21:43:46.802393+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:29.902606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.902606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":601,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802284+0000","last_change":"2024-03-26T21:41:06.760581+0000","last_active":"2024-03-26T21:43:46.802284+0000","last_peered":"2024-03-26T21:43:46.802284+0000","last_clean":"2024-03-26T21:43:46.802284+0000","last_became_active":"2024-03-26T21:41:06.689846+0000","last_became_peered":"2024-03-26T21:41:06.689846+0000","last_unstale":"2024-03-26T21:43:46.802284+0000","last_undegraded":"2024-03-26T21:43:46.802284+0000","last_fullsized":"2024-03-26T21:43:46.802284+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-26T21:40:38.654652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.654652+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":534,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807334+0000","last_change":"2024-03-26T21:43:06.349697+0000","last_active":"2024-03-26T21:43:46.807334+0000","last_peered":"2024-03-26T21:43:46.807334+0000","last_clean":"2024-03-26T21:43:46.807334+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:43:46.807334+0000","last_undegraded":"2024-03-26T21:43:46.807334+0000","last_fullsized":"2024-03-26T21:43:46.807334+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:06.349429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.349429+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:13.011189+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":839,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.883559+0000","last_change":"2024-03-26T21:41:30.072506+0000","last_active":"2024-03-26T21:43:44.883559+0000","last_peered":"2024-03-26T21:43:44.883559+0000","last_clean":"2024-03-26T21:43:44.883559+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:43:44.883559+0000","last_undegraded":"2024-03-26T21:43:44.883559+0000","last_fullsized":"2024-03-26T21:43:44.883559+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:41:30.072374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:30.072374+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":652,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801824+0000","last_change":"2024-03-26T21:41:06.760645+0000","last_active":"2024-03-26T21:43:46.801824+0000","last_peered":"2024-03-26T21:43:46.801824+0000","last_clean":"2024-03-26T21:43:46.801824+0000","last_became_active":"2024-03-26T21:41:06.699136+0000","last_became_peered":"2024-03-26T21:41:06.699136+0000","last_unstale":"2024-03-26T21:43:46.801824+0000","last_undegraded":"2024-03-26T21:43:46.801824+0000","last_fullsized":"2024-03-26T21:43:46.801824+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:40:23.862904+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.862904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":665,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805609+0000","last_change":"2024-03-26T21:43:16.803353+0000","last_active":"2024-03-26T21:43:46.805609+0000","last_peered":"2024-03-26T21:43:46.805609+0000","last_clean":"2024-03-26T21:43:46.805609+0000","last_became_active":"2024-03-26T21:43:16.797672+0000","last_became_peered":"2024-03-26T21:43:16.797672+0000","last_unstale":"2024-03-26T21:43:46.805609+0000","last_undegraded":"2024-03-26T21:43:46.805609+0000","last_fullsized":"2024-03-26T21:43:46.805609+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:39:33.305513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.305513+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802222+0000","last_change":"2024-03-26T21:40:16.025952+0000","last_active":"2024-03-26T21:43:46.802222+0000","last_peered":"2024-03-26T21:43:46.802222+0000","last_clean":"2024-03-26T21:43:46.802222+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:43:46.802222+0000","last_undegraded":"2024-03-26T21:43:46.802222+0000","last_fullsized":"2024-03-26T21:43:46.802222+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.025838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.025838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":605,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805868+0000","last_change":"2024-03-26T21:42:59.198607+0000","last_active":"2024-03-26T21:43:46.805868+0000","last_peered":"2024-03-26T21:43:46.805868+0000","last_clean":"2024-03-26T21:43:46.805868+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:43:46.805868+0000","last_undegraded":"2024-03-26T21:43:46.805868+0000","last_fullsized":"2024-03-26T21:43:46.805868+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.198481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.198481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.882421+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":153,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806881+0000","last_change":"2024-03-26T21:41:06.684801+0000","last_active":"2024-03-26T21:43:46.806881+0000","last_peered":"2024-03-26T21:43:46.806881+0000","last_clean":"2024-03-26T21:43:46.806881+0000","last_became_active":"2024-03-26T21:41:06.683627+0000","last_became_peered":"2024-03-26T21:41:06.683627+0000","last_unstale":"2024-03-26T21:43:46.806881+0000","last_undegraded":"2024-03-26T21:43:46.806881+0000","last_fullsized":"2024-03-26T21:43:46.806881+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:24.208179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.208179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":658,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.883258+0000","last_change":"2024-03-26T21:42:49.300575+0000","last_active":"2024-03-26T21:43:44.883258+0000","last_peered":"2024-03-26T21:43:44.883258+0000","last_clean":"2024-03-26T21:43:44.883258+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:43:44.883258+0000","last_undegraded":"2024-03-26T21:43:44.883258+0000","last_fullsized":"2024-03-26T21:43:44.883258+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:42:49.300477+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.300477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.397719+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":531,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806977+0000","last_change":"2024-03-26T21:41:25.602252+0000","last_active":"2024-03-26T21:43:46.806977+0000","last_peered":"2024-03-26T21:43:46.806977+0000","last_clean":"2024-03-26T21:43:46.806977+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:43:46.806977+0000","last_undegraded":"2024-03-26T21:43:46.806977+0000","last_fullsized":"2024-03-26T21:43:46.806977+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:54.032885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:38:54.032885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":639,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.881168+0000","last_change":"2024-03-26T21:43:06.168376+0000","last_active":"2024-03-26T21:43:44.881168+0000","last_peered":"2024-03-26T21:43:44.881168+0000","last_clean":"2024-03-26T21:43:44.881168+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:43:44.881168+0000","last_undegraded":"2024-03-26T21:43:44.881168+0000","last_fullsized":"2024-03-26T21:43:44.881168+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.168150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.168150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:11.341635+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":609,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806866+0000","last_change":"2024-03-26T21:41:06.674643+0000","last_active":"2024-03-26T21:43:46.806866+0000","last_peered":"2024-03-26T21:43:46.806866+0000","last_clean":"2024-03-26T21:43:46.806866+0000","last_became_active":"2024-03-26T21:41:06.674252+0000","last_became_peered":"2024-03-26T21:41:06.674252+0000","last_unstale":"2024-03-26T21:43:46.806866+0000","last_undegraded":"2024-03-26T21:43:46.806866+0000","last_fullsized":"2024-03-26T21:43:46.806866+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:29.294907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.294907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":546,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801888+0000","last_change":"2024-03-26T21:41:06.726632+0000","last_active":"2024-03-26T21:43:46.801888+0000","last_peered":"2024-03-26T21:43:46.801888+0000","last_clean":"2024-03-26T21:43:46.801888+0000","last_became_active":"2024-03-26T21:41:06.691526+0000","last_became_peered":"2024-03-26T21:41:06.691526+0000","last_unstale":"2024-03-26T21:43:46.801888+0000","last_undegraded":"2024-03-26T21:43:46.801888+0000","last_fullsized":"2024-03-26T21:43:46.801888+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:40:48.743775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.743775+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":154,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805047+0000","last_change":"2024-03-26T21:41:06.682123+0000","last_active":"2024-03-26T21:43:46.805047+0000","last_peered":"2024-03-26T21:43:46.805047+0000","last_clean":"2024-03-26T21:43:46.805047+0000","last_became_active":"2024-03-26T21:41:06.681798+0000","last_became_peered":"2024-03-26T21:41:06.681798+0000","last_unstale":"2024-03-26T21:43:46.805047+0000","last_undegraded":"2024-03-26T21:43:46.805047+0000","last_fullsized":"2024-03-26T21:43:46.805047+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:36.268946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:39:36.268946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":531,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807307+0000","last_change":"2024-03-26T21:42:58.360038+0000","last_active":"2024-03-26T21:43:46.807307+0000","last_peered":"2024-03-26T21:43:46.807307+0000","last_clean":"2024-03-26T21:43:46.807307+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:43:46.807307+0000","last_undegraded":"2024-03-26T21:43:46.807307+0000","last_fullsized":"2024-03-26T21:43:46.807307+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.359906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.359906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.193584+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":619,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810565+0000","last_change":"2024-03-26T21:42:50.211297+0000","last_active":"2024-03-26T21:43:46.810565+0000","last_peered":"2024-03-26T21:43:46.810565+0000","last_clean":"2024-03-26T21:43:46.810565+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:43:46.810565+0000","last_undegraded":"2024-03-26T21:43:46.810565+0000","last_fullsized":"2024-03-26T21:43:46.810565+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:50.211214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:42:50.211214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.540886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":536,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809085+0000","last_change":"2024-03-26T21:41:26.595088+0000","last_active":"2024-03-26T21:43:46.809085+0000","last_peered":"2024-03-26T21:43:46.809085+0000","last_clean":"2024-03-26T21:43:46.809085+0000","last_became_active":"2024-03-26T21:41:26.594715+0000","last_became_peered":"2024-03-26T21:41:26.594715+0000","last_unstale":"2024-03-26T21:43:46.809085+0000","last_undegraded":"2024-03-26T21:43:46.809085+0000","last_fullsized":"2024-03-26T21:43:46.809085+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:29.152531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.152531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":591,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806588+0000","last_change":"2024-03-26T21:40:17.751277+0000","last_active":"2024-03-26T21:43:46.806588+0000","last_peered":"2024-03-26T21:43:46.806588+0000","last_clean":"2024-03-26T21:43:46.806588+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:43:46.806588+0000","last_undegraded":"2024-03-26T21:43:46.806588+0000","last_fullsized":"2024-03-26T21:43:46.806588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.748978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.748978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":161,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805178+0000","last_change":"2024-03-26T21:43:00.150777+0000","last_active":"2024-03-26T21:43:46.805178+0000","last_peered":"2024-03-26T21:43:46.805178+0000","last_clean":"2024-03-26T21:43:46.805178+0000","last_became_active":"2024-03-26T21:41:06.669878+0000","last_became_peered":"2024-03-26T21:41:06.669878+0000","last_unstale":"2024-03-26T21:43:46.805178+0000","last_undegraded":"2024-03-26T21:43:46.805178+0000","last_fullsized":"2024-03-26T21:43:46.805178+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.150652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.150652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.910010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":540,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808355+0000","last_change":"2024-03-26T21:41:26.681701+0000","last_active":"2024-03-26T21:43:46.808355+0000","last_peered":"2024-03-26T21:43:46.808355+0000","last_clean":"2024-03-26T21:43:46.808355+0000","last_became_active":"2024-03-26T21:41:26.681323+0000","last_became_peered":"2024-03-26T21:41:26.681323+0000","last_unstale":"2024-03-26T21:43:46.808355+0000","last_undegraded":"2024-03-26T21:43:46.808355+0000","last_fullsized":"2024-03-26T21:43:46.808355+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:46.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:38:46.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":636,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802831+0000","last_change":"2024-03-26T21:39:44.903744+0000","last_active":"2024-03-26T21:43:46.802831+0000","last_peered":"2024-03-26T21:43:46.802831+0000","last_clean":"2024-03-26T21:43:46.802831+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:43:46.802831+0000","last_undegraded":"2024-03-26T21:43:46.802831+0000","last_fullsized":"2024-03-26T21:43:46.802831+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:44.903549+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:39:44.903549+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":154,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806211+0000","last_change":"2024-03-26T21:41:06.692468+0000","last_active":"2024-03-26T21:43:46.806211+0000","last_peered":"2024-03-26T21:43:46.806211+0000","last_clean":"2024-03-26T21:43:46.806211+0000","last_became_active":"2024-03-26T21:41:06.689995+0000","last_became_peered":"2024-03-26T21:41:06.689995+0000","last_unstale":"2024-03-26T21:43:46.806211+0000","last_undegraded":"2024-03-26T21:43:46.806211+0000","last_fullsized":"2024-03-26T21:43:46.806211+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.810420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802641+0000","last_change":"2024-03-26T21:41:26.614343+0000","last_active":"2024-03-26T21:43:46.802641+0000","last_peered":"2024-03-26T21:43:46.802641+0000","last_clean":"2024-03-26T21:43:46.802641+0000","last_became_active":"2024-03-26T21:41:26.612029+0000","last_became_peered":"2024-03-26T21:41:26.612029+0000","last_unstale":"2024-03-26T21:43:46.802641+0000","last_undegraded":"2024-03-26T21:43:46.802641+0000","last_fullsized":"2024-03-26T21:43:46.802641+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:36.141522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.141522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":308,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803024+0000","last_change":"2024-03-26T21:41:31.911664+0000","last_active":"2024-03-26T21:43:46.803024+0000","last_peered":"2024-03-26T21:43:46.803024+0000","last_clean":"2024-03-26T21:43:46.803024+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:43:46.803024+0000","last_undegraded":"2024-03-26T21:43:46.803024+0000","last_fullsized":"2024-03-26T21:43:46.803024+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:31.911237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:31.911237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":331,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802118+0000","last_change":"2024-03-26T21:41:44.918112+0000","last_active":"2024-03-26T21:43:46.802118+0000","last_peered":"2024-03-26T21:43:46.802118+0000","last_clean":"2024-03-26T21:43:46.802118+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:43:46.802118+0000","last_undegraded":"2024-03-26T21:43:46.802118+0000","last_fullsized":"2024-03-26T21:43:46.802118+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:44.917970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:41:44.917970+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":609,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.185401+0000","last_change":"2024-03-26T21:43:48.185447+0000","last_active":"2024-03-26T21:43:48.185401+0000","last_peered":"2024-03-26T21:43:48.185401+0000","last_clean":"2024-03-26T21:43:48.185401+0000","last_became_active":"2024-03-26T21:41:26.607685+0000","last_became_peered":"2024-03-26T21:41:26.607685+0000","last_unstale":"2024-03-26T21:43:48.185401+0000","last_undegraded":"2024-03-26T21:43:48.185401+0000","last_fullsized":"2024-03-26T21:43:48.185401+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:48.185306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:48.185306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:13.252836+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":794,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810081+0000","last_change":"2024-03-26T21:43:08.196513+0000","last_active":"2024-03-26T21:43:46.810081+0000","last_peered":"2024-03-26T21:43:46.810081+0000","last_clean":"2024-03-26T21:43:46.810081+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:43:46.810081+0000","last_undegraded":"2024-03-26T21:43:46.810081+0000","last_fullsized":"2024-03-26T21:43:46.810081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:43:08.196428+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.196428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.141469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":549,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807690+0000","last_change":"2024-03-26T21:43:45.070796+0000","last_active":"2024-03-26T21:43:46.807690+0000","last_peered":"2024-03-26T21:43:46.807690+0000","last_clean":"2024-03-26T21:43:46.807690+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:43:46.807690+0000","last_undegraded":"2024-03-26T21:43:46.807690+0000","last_fullsized":"2024-03-26T21:43:46.807690+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:45.070590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:45.070590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:04.006594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":587,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802472+0000","last_change":"2024-03-26T21:42:58.972473+0000","last_active":"2024-03-26T21:43:46.802472+0000","last_peered":"2024-03-26T21:43:46.802472+0000","last_clean":"2024-03-26T21:43:46.802472+0000","last_became_active":"2024-03-26T21:41:26.612486+0000","last_became_peered":"2024-03-26T21:41:26.612486+0000","last_unstale":"2024-03-26T21:43:46.802472+0000","last_undegraded":"2024-03-26T21:43:46.802472+0000","last_fullsized":"2024-03-26T21:43:46.802472+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:42:58.972320+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.972320+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.479102+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":307,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802876+0000","last_change":"2024-03-26T21:42:54.787575+0000","last_active":"2024-03-26T21:43:46.802876+0000","last_peered":"2024-03-26T21:43:46.802876+0000","last_clean":"2024-03-26T21:43:46.802876+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:43:46.802876+0000","last_undegraded":"2024-03-26T21:43:46.802876+0000","last_fullsized":"2024-03-26T21:43:46.802876+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.787464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.787464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.976476+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"278'12","reported_seq":195,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808611+0000","last_change":"2024-03-26T21:41:26.613897+0000","last_active":"2024-03-26T21:43:46.808611+0000","last_peered":"2024-03-26T21:43:46.808611+0000","last_clean":"2024-03-26T21:43:46.808611+0000","last_became_active":"2024-03-26T21:41:26.612842+0000","last_became_peered":"2024-03-26T21:41:26.612842+0000","last_unstale":"2024-03-26T21:43:46.808611+0000","last_undegraded":"2024-03-26T21:43:46.808611+0000","last_fullsized":"2024-03-26T21:43:46.808611+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:39:27.133215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:27.133215+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":616,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805226+0000","last_change":"2024-03-26T21:43:07.208656+0000","last_active":"2024-03-26T21:43:46.805226+0000","last_peered":"2024-03-26T21:43:46.805226+0000","last_clean":"2024-03-26T21:43:46.805226+0000","last_became_active":"2024-03-26T21:41:26.607687+0000","last_became_peered":"2024-03-26T21:41:26.607687+0000","last_unstale":"2024-03-26T21:43:46.805226+0000","last_undegraded":"2024-03-26T21:43:46.805226+0000","last_fullsized":"2024-03-26T21:43:46.805226+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:07.208520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:07.208520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:31.173304+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":658,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810244+0000","last_change":"2024-03-26T21:42:49.162778+0000","last_active":"2024-03-26T21:43:46.810244+0000","last_peered":"2024-03-26T21:43:46.810244+0000","last_clean":"2024-03-26T21:43:46.810244+0000","last_became_active":"2024-03-26T21:41:26.596451+0000","last_became_peered":"2024-03-26T21:41:26.596451+0000","last_unstale":"2024-03-26T21:43:46.810244+0000","last_undegraded":"2024-03-26T21:43:46.810244+0000","last_fullsized":"2024-03-26T21:43:46.810244+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:49.162638+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.162638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:15.859076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":629,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810043+0000","last_change":"2024-03-26T21:42:44.266704+0000","last_active":"2024-03-26T21:43:46.810043+0000","last_peered":"2024-03-26T21:43:46.810043+0000","last_clean":"2024-03-26T21:43:46.810043+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:43:46.810043+0000","last_undegraded":"2024-03-26T21:43:46.810043+0000","last_fullsized":"2024-03-26T21:43:46.810043+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:44.266582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:44.266582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:56.439301+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":559,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803133+0000","last_change":"2024-03-26T21:42:56.932427+0000","last_active":"2024-03-26T21:43:46.803133+0000","last_peered":"2024-03-26T21:43:46.803133+0000","last_clean":"2024-03-26T21:43:46.803133+0000","last_became_active":"2024-03-26T21:41:26.610882+0000","last_became_peered":"2024-03-26T21:41:26.610882+0000","last_unstale":"2024-03-26T21:43:46.803133+0000","last_undegraded":"2024-03-26T21:43:46.803133+0000","last_fullsized":"2024-03-26T21:43:46.803133+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:04.687295+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":576,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808846+0000","last_change":"2024-03-26T21:41:25.623784+0000","last_active":"2024-03-26T21:43:46.808846+0000","last_peered":"2024-03-26T21:43:46.808846+0000","last_clean":"2024-03-26T21:43:46.808846+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:43:46.808846+0000","last_undegraded":"2024-03-26T21:43:46.808846+0000","last_fullsized":"2024-03-26T21:43:46.808846+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:42.962120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:42.962120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":651,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805710+0000","last_change":"2024-03-26T21:42:51.070243+0000","last_active":"2024-03-26T21:43:46.805710+0000","last_peered":"2024-03-26T21:43:46.805710+0000","last_clean":"2024-03-26T21:43:46.805710+0000","last_became_active":"2024-03-26T21:41:26.614253+0000","last_became_peered":"2024-03-26T21:41:26.614253+0000","last_unstale":"2024-03-26T21:43:46.805710+0000","last_undegraded":"2024-03-26T21:43:46.805710+0000","last_fullsized":"2024-03-26T21:43:46.805710+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:51.070126+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.070126+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:00.147180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":531,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808271+0000","last_change":"2024-03-26T21:41:25.622629+0000","last_active":"2024-03-26T21:43:46.808271+0000","last_peered":"2024-03-26T21:43:46.808271+0000","last_clean":"2024-03-26T21:43:46.808271+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:43:46.808271+0000","last_undegraded":"2024-03-26T21:43:46.808271+0000","last_fullsized":"2024-03-26T21:43:46.808271+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:50.068375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:38:50.068375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":608,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810764+0000","last_change":"2024-03-26T21:41:06.691730+0000","last_active":"2024-03-26T21:43:46.810764+0000","last_peered":"2024-03-26T21:43:46.810764+0000","last_clean":"2024-03-26T21:43:46.810764+0000","last_became_active":"2024-03-26T21:41:06.691045+0000","last_became_peered":"2024-03-26T21:41:06.691045+0000","last_unstale":"2024-03-26T21:43:46.810764+0000","last_undegraded":"2024-03-26T21:43:46.810764+0000","last_fullsized":"2024-03-26T21:43:46.810764+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:34.942806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:34.942806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":625,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:45.525265+0000","last_change":"2024-03-26T21:43:45.525304+0000","last_active":"2024-03-26T21:43:45.525265+0000","last_peered":"2024-03-26T21:43:45.525265+0000","last_clean":"2024-03-26T21:43:45.525265+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:43:45.525265+0000","last_undegraded":"2024-03-26T21:43:45.525265+0000","last_fullsized":"2024-03-26T21:43:45.525265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:45.525163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:45.525163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:06.542305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":544,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808712+0000","last_change":"2024-03-26T21:42:51.267747+0000","last_active":"2024-03-26T21:43:46.808712+0000","last_peered":"2024-03-26T21:43:46.808712+0000","last_clean":"2024-03-26T21:43:46.808712+0000","last_became_active":"2024-03-26T21:41:26.612479+0000","last_became_peered":"2024-03-26T21:41:26.612479+0000","last_unstale":"2024-03-26T21:43:46.808712+0000","last_undegraded":"2024-03-26T21:43:46.808712+0000","last_fullsized":"2024-03-26T21:43:46.808712+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:42:51.267662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.267662+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:10.904667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"340'1093","reported_seq":1125,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:47.023839+0000","last_change":"2024-03-26T21:43:28.830391+0000","last_active":"2024-03-26T21:43:47.023839+0000","last_peered":"2024-03-26T21:43:47.023839+0000","last_clean":"2024-03-26T21:43:47.023839+0000","last_became_active":"2024-03-26T21:43:28.829928+0000","last_became_peered":"2024-03-26T21:43:28.829928+0000","last_unstale":"2024-03-26T21:43:47.023839+0000","last_undegraded":"2024-03-26T21:43:47.023839+0000","last_fullsized":"2024-03-26T21:43:47.023839+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:55.393644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630848,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1093,"num_write_kb":69952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":590,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810215+0000","last_change":"2024-03-26T21:40:25.046799+0000","last_active":"2024-03-26T21:43:46.810215+0000","last_peered":"2024-03-26T21:43:46.810215+0000","last_clean":"2024-03-26T21:43:46.810215+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:43:46.810215+0000","last_undegraded":"2024-03-26T21:43:46.810215+0000","last_fullsized":"2024-03-26T21:43:46.810215+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:25.046675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:25.046675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":595,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808566+0000","last_change":"2024-03-26T21:41:25.611205+0000","last_active":"2024-03-26T21:43:46.808566+0000","last_peered":"2024-03-26T21:43:46.808566+0000","last_clean":"2024-03-26T21:43:46.808566+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:43:46.808566+0000","last_undegraded":"2024-03-26T21:43:46.808566+0000","last_fullsized":"2024-03-26T21:43:46.808566+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:39:22.057217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:22.057217+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":658,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809286+0000","last_change":"2024-03-26T21:42:52.208110+0000","last_active":"2024-03-26T21:43:46.809286+0000","last_peered":"2024-03-26T21:43:46.809286+0000","last_clean":"2024-03-26T21:43:46.809286+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:43:46.809286+0000","last_undegraded":"2024-03-26T21:43:46.809286+0000","last_fullsized":"2024-03-26T21:43:46.809286+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:52.207964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.207964+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.739899+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":732,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808486+0000","last_change":"2024-03-26T21:41:25.624173+0000","last_active":"2024-03-26T21:43:46.808486+0000","last_peered":"2024-03-26T21:43:46.808486+0000","last_clean":"2024-03-26T21:43:46.808486+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:43:46.808486+0000","last_undegraded":"2024-03-26T21:43:46.808486+0000","last_fullsized":"2024-03-26T21:43:46.808486+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:39:25.023169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.023169+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"339'1044","reported_seq":1074,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.771897+0000","last_change":"2024-03-26T21:43:28.831648+0000","last_active":"2024-03-26T21:43:46.771897+0000","last_peered":"2024-03-26T21:43:46.771897+0000","last_clean":"2024-03-26T21:43:46.771897+0000","last_became_active":"2024-03-26T21:43:28.831305+0000","last_became_peered":"2024-03-26T21:43:28.831305+0000","last_unstale":"2024-03-26T21:43:46.771897+0000","last_undegraded":"2024-03-26T21:43:46.771897+0000","last_fullsized":"2024-03-26T21:43:46.771897+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:48.367675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":66816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":147,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807320+0000","last_change":"2024-03-26T21:42:56.446256+0000","last_active":"2024-03-26T21:43:46.807320+0000","last_peered":"2024-03-26T21:43:46.807320+0000","last_clean":"2024-03-26T21:43:46.807320+0000","last_became_active":"2024-03-26T21:41:26.600529+0000","last_became_peered":"2024-03-26T21:41:26.600529+0000","last_unstale":"2024-03-26T21:43:46.807320+0000","last_undegraded":"2024-03-26T21:43:46.807320+0000","last_fullsized":"2024-03-26T21:43:46.807320+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.446156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.446156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:08.661338+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":586,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808313+0000","last_change":"2024-03-26T21:41:25.610560+0000","last_active":"2024-03-26T21:43:46.808313+0000","last_peered":"2024-03-26T21:43:46.808313+0000","last_clean":"2024-03-26T21:43:46.808313+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:43:46.808313+0000","last_undegraded":"2024-03-26T21:43:46.808313+0000","last_fullsized":"2024-03-26T21:43:46.808313+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:39:32.049629+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:39:32.049629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":344,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802927+0000","last_change":"2024-03-26T21:41:43.970623+0000","last_active":"2024-03-26T21:43:46.802927+0000","last_peered":"2024-03-26T21:43:46.802927+0000","last_clean":"2024-03-26T21:43:46.802927+0000","last_became_active":"2024-03-26T21:41:06.689727+0000","last_became_peered":"2024-03-26T21:41:06.689727+0000","last_unstale":"2024-03-26T21:43:46.802927+0000","last_undegraded":"2024-03-26T21:43:46.802927+0000","last_fullsized":"2024-03-26T21:43:46.802927+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.970479+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":619,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809385+0000","last_change":"2024-03-26T21:42:54.268952+0000","last_active":"2024-03-26T21:43:46.809385+0000","last_peered":"2024-03-26T21:43:46.809385+0000","last_clean":"2024-03-26T21:43:46.809385+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:43:46.809385+0000","last_undegraded":"2024-03-26T21:43:46.809385+0000","last_fullsized":"2024-03-26T21:43:46.809385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.268871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:03.452361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"340'1358","reported_seq":1390,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.612474+0000","last_change":"2024-03-26T21:43:28.830699+0000","last_active":"2024-03-26T21:43:48.612474+0000","last_peered":"2024-03-26T21:43:48.612474+0000","last_clean":"2024-03-26T21:43:48.612474+0000","last_became_active":"2024-03-26T21:43:28.830394+0000","last_became_peered":"2024-03-26T21:43:28.830394+0000","last_unstale":"2024-03-26T21:43:48.612474+0000","last_undegraded":"2024-03-26T21:43:48.612474+0000","last_fullsized":"2024-03-26T21:43:48.612474+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:47.192469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88997888,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":86912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":523,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806701+0000","last_change":"2024-03-26T21:43:05.373915+0000","last_active":"2024-03-26T21:43:46.806701+0000","last_peered":"2024-03-26T21:43:46.806701+0000","last_clean":"2024-03-26T21:43:46.806701+0000","last_became_active":"2024-03-26T21:41:26.596674+0000","last_became_peered":"2024-03-26T21:41:26.596674+0000","last_unstale":"2024-03-26T21:43:46.806701+0000","last_undegraded":"2024-03-26T21:43:46.806701+0000","last_fullsized":"2024-03-26T21:43:46.806701+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.373726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.373726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:26.697791+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":627,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810023+0000","last_change":"2024-03-26T21:43:01.366930+0000","last_active":"2024-03-26T21:43:46.810023+0000","last_peered":"2024-03-26T21:43:46.810023+0000","last_clean":"2024-03-26T21:43:46.810023+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:43:46.810023+0000","last_undegraded":"2024-03-26T21:43:46.810023+0000","last_fullsized":"2024-03-26T21:43:46.810023+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.366789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.758642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":556,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808459+0000","last_change":"2024-03-26T21:41:25.626666+0000","last_active":"2024-03-26T21:43:46.808459+0000","last_peered":"2024-03-26T21:43:46.808459+0000","last_clean":"2024-03-26T21:43:46.808459+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:43:46.808459+0000","last_undegraded":"2024-03-26T21:43:46.808459+0000","last_fullsized":"2024-03-26T21:43:46.808459+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:23.044612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:23.044612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":687,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.882782+0000","last_change":"2024-03-26T21:41:36.095640+0000","last_active":"2024-03-26T21:43:44.882782+0000","last_peered":"2024-03-26T21:43:44.882782+0000","last_clean":"2024-03-26T21:43:44.882782+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:43:44.882782+0000","last_undegraded":"2024-03-26T21:43:44.882782+0000","last_fullsized":"2024-03-26T21:43:44.882782+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:41:36.095502+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:41:36.095502+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"340'1069","reported_seq":1101,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.931079+0000","last_change":"2024-03-26T21:43:28.830442+0000","last_active":"2024-03-26T21:43:46.931079+0000","last_peered":"2024-03-26T21:43:46.931079+0000","last_clean":"2024-03-26T21:43:46.931079+0000","last_became_active":"2024-03-26T21:43:28.830034+0000","last_became_peered":"2024-03-26T21:43:28.830034+0000","last_unstale":"2024-03-26T21:43:46.931079+0000","last_undegraded":"2024-03-26T21:43:46.931079+0000","last_fullsized":"2024-03-26T21:43:46.931079+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:45.131239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70057984,"num_objects":1069,"num_object_clones":0,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1069,"num_write_kb":68416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":592,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806393+0000","last_change":"2024-03-26T21:41:25.611985+0000","last_active":"2024-03-26T21:43:46.806393+0000","last_peered":"2024-03-26T21:43:46.806393+0000","last_clean":"2024-03-26T21:43:46.806393+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:43:46.806393+0000","last_undegraded":"2024-03-26T21:43:46.806393+0000","last_fullsized":"2024-03-26T21:43:46.806393+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:38:41.522724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:38:41.522724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":638,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809343+0000","last_change":"2024-03-26T21:42:55.277471+0000","last_active":"2024-03-26T21:43:46.809343+0000","last_peered":"2024-03-26T21:43:46.809343+0000","last_clean":"2024-03-26T21:43:46.809343+0000","last_became_active":"2024-03-26T21:41:06.673129+0000","last_became_peered":"2024-03-26T21:41:06.673129+0000","last_unstale":"2024-03-26T21:43:46.809343+0000","last_undegraded":"2024-03-26T21:43:46.809343+0000","last_fullsized":"2024-03-26T21:43:46.809343+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.277361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:12.168390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809932+0000","last_change":"2024-03-26T21:43:46.292146+0000","last_active":"2024-03-26T21:43:46.809932+0000","last_peered":"2024-03-26T21:43:46.809932+0000","last_clean":"2024-03-26T21:43:46.809932+0000","last_became_active":"2024-03-26T21:41:06.690153+0000","last_became_peered":"2024-03-26T21:41:06.690153+0000","last_unstale":"2024-03-26T21:43:46.809932+0000","last_undegraded":"2024-03-26T21:43:46.809932+0000","last_fullsized":"2024-03-26T21:43:46.809932+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:46.291979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:43:46.291979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:57.838621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":649,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805520+0000","last_change":"2024-03-26T21:39:46.369958+0000","last_active":"2024-03-26T21:43:46.805520+0000","last_peered":"2024-03-26T21:43:46.805520+0000","last_clean":"2024-03-26T21:43:46.805520+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:43:46.805520+0000","last_undegraded":"2024-03-26T21:43:46.805520+0000","last_fullsized":"2024-03-26T21:43:46.805520+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:39:46.369461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:46.369461+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"339'1153","reported_seq":1182,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.759036+0000","last_change":"2024-03-26T21:43:28.830559+0000","last_active":"2024-03-26T21:43:46.759036+0000","last_peered":"2024-03-26T21:43:46.759036+0000","last_clean":"2024-03-26T21:43:46.759036+0000","last_became_active":"2024-03-26T21:43:28.830040+0000","last_became_peered":"2024-03-26T21:43:28.830040+0000","last_unstale":"2024-03-26T21:43:46.759036+0000","last_undegraded":"2024-03-26T21:43:46.759036+0000","last_fullsized":"2024-03-26T21:43:46.759036+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1153,"log_dups_size":0,"ondisk_log_size":1153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:43.194190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1153,"num_write_kb":73792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":577,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802836+0000","last_change":"2024-03-26T21:41:27.901875+0000","last_active":"2024-03-26T21:43:46.802836+0000","last_peered":"2024-03-26T21:43:46.802836+0000","last_clean":"2024-03-26T21:43:46.802836+0000","last_became_active":"2024-03-26T21:41:26.593537+0000","last_became_peered":"2024-03-26T21:41:26.593537+0000","last_unstale":"2024-03-26T21:43:46.802836+0000","last_undegraded":"2024-03-26T21:43:46.802836+0000","last_fullsized":"2024-03-26T21:43:46.802836+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:27.901800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:41:27.901800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":621,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809924+0000","last_change":"2024-03-26T21:40:36.901922+0000","last_active":"2024-03-26T21:43:46.809924+0000","last_peered":"2024-03-26T21:43:46.809924+0000","last_clean":"2024-03-26T21:43:46.809924+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:43:46.809924+0000","last_undegraded":"2024-03-26T21:43:46.809924+0000","last_fullsized":"2024-03-26T21:43:46.809924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:40:36.901814+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.901814+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809795+0000","last_change":"2024-03-26T21:43:16.796612+0000","last_active":"2024-03-26T21:43:46.809795+0000","last_peered":"2024-03-26T21:43:46.809795+0000","last_clean":"2024-03-26T21:43:46.809795+0000","last_became_active":"2024-03-26T21:43:16.796261+0000","last_became_peered":"2024-03-26T21:43:16.796261+0000","last_unstale":"2024-03-26T21:43:46.809795+0000","last_undegraded":"2024-03-26T21:43:46.809795+0000","last_fullsized":"2024-03-26T21:43:46.809795+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.930550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.930550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"270'42","reported_seq":839,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809563+0000","last_change":"2024-03-26T21:42:55.163748+0000","last_active":"2024-03-26T21:43:46.809563+0000","last_peered":"2024-03-26T21:43:46.809563+0000","last_clean":"2024-03-26T21:43:46.809563+0000","last_became_active":"2024-03-26T21:41:06.684878+0000","last_became_peered":"2024-03-26T21:41:06.684878+0000","last_unstale":"2024-03-26T21:43:46.809563+0000","last_undegraded":"2024-03-26T21:43:46.809563+0000","last_fullsized":"2024-03-26T21:43:46.809563+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'42","last_scrub_stamp":"2024-03-26T21:42:55.163584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.163584+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:22.239591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805745+0000","last_change":"2024-03-26T21:40:38.776211+0000","last_active":"2024-03-26T21:43:46.805745+0000","last_peered":"2024-03-26T21:43:46.805745+0000","last_clean":"2024-03-26T21:43:46.805745+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:43:46.805745+0000","last_undegraded":"2024-03-26T21:43:46.805745+0000","last_fullsized":"2024-03-26T21:43:46.805745+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:38.774895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.774895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"340'1150","reported_seq":1182,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:47.010316+0000","last_change":"2024-03-26T21:43:28.830002+0000","last_active":"2024-03-26T21:43:47.010316+0000","last_peered":"2024-03-26T21:43:47.010316+0000","last_clean":"2024-03-26T21:43:47.010316+0000","last_became_active":"2024-03-26T21:43:28.829410+0000","last_became_peered":"2024-03-26T21:43:28.829410+0000","last_unstale":"2024-03-26T21:43:47.010316+0000","last_undegraded":"2024-03-26T21:43:47.010316+0000","last_fullsized":"2024-03-26T21:43:47.010316+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:37.094703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":3450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1150,"num_write_kb":73600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":515,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808140+0000","last_change":"2024-03-26T21:41:26.597095+0000","last_active":"2024-03-26T21:43:46.808140+0000","last_peered":"2024-03-26T21:43:46.808140+0000","last_clean":"2024-03-26T21:43:46.808140+0000","last_became_active":"2024-03-26T21:41:26.596693+0000","last_became_peered":"2024-03-26T21:41:26.596693+0000","last_unstale":"2024-03-26T21:43:46.808140+0000","last_undegraded":"2024-03-26T21:43:46.808140+0000","last_fullsized":"2024-03-26T21:43:46.808140+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:23.213806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:41:23.213806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":628,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801795+0000","last_change":"2024-03-26T21:40:46.085237+0000","last_active":"2024-03-26T21:43:46.801795+0000","last_peered":"2024-03-26T21:43:46.801795+0000","last_clean":"2024-03-26T21:43:46.801795+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:43:46.801795+0000","last_undegraded":"2024-03-26T21:43:46.801795+0000","last_fullsized":"2024-03-26T21:43:46.801795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:46.085013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:40:46.085013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":633,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809369+0000","last_change":"2024-03-26T21:42:49.297990+0000","last_active":"2024-03-26T21:43:46.809369+0000","last_peered":"2024-03-26T21:43:46.809369+0000","last_clean":"2024-03-26T21:43:46.809369+0000","last_became_active":"2024-03-26T21:41:06.690996+0000","last_became_peered":"2024-03-26T21:41:06.690996+0000","last_unstale":"2024-03-26T21:43:46.809369+0000","last_undegraded":"2024-03-26T21:43:46.809369+0000","last_fullsized":"2024-03-26T21:43:46.809369+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:49.297842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.297842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:50.237836+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":654,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803590+0000","last_change":"2024-03-26T21:40:17.856788+0000","last_active":"2024-03-26T21:43:46.803590+0000","last_peered":"2024-03-26T21:43:46.803590+0000","last_clean":"2024-03-26T21:43:46.803590+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:43:46.803590+0000","last_undegraded":"2024-03-26T21:43:46.803590+0000","last_fullsized":"2024-03-26T21:43:46.803590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:40:17.856643+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.856643+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":621,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810420+0000","last_change":"2024-03-26T21:43:43.406078+0000","last_active":"2024-03-26T21:43:46.810420+0000","last_peered":"2024-03-26T21:43:46.810420+0000","last_clean":"2024-03-26T21:43:46.810420+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:43:46.810420+0000","last_undegraded":"2024-03-26T21:43:46.810420+0000","last_fullsized":"2024-03-26T21:43:46.810420+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:43.405952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:43:43.405952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:09.910118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"339'1134","reported_seq":1164,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.775525+0000","last_change":"2024-03-26T21:43:28.828609+0000","last_active":"2024-03-26T21:43:46.775525+0000","last_peered":"2024-03-26T21:43:46.775525+0000","last_clean":"2024-03-26T21:43:46.775525+0000","last_became_active":"2024-03-26T21:43:28.827990+0000","last_became_peered":"2024-03-26T21:43:28.827990+0000","last_unstale":"2024-03-26T21:43:46.775525+0000","last_undegraded":"2024-03-26T21:43:46.775525+0000","last_fullsized":"2024-03-26T21:43:46.775525+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:41.294519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810581+0000","last_change":"2024-03-26T21:41:06.692795+0000","last_active":"2024-03-26T21:43:46.810581+0000","last_peered":"2024-03-26T21:43:46.810581+0000","last_clean":"2024-03-26T21:43:46.810581+0000","last_became_active":"2024-03-26T21:41:06.691293+0000","last_became_peered":"2024-03-26T21:41:06.691293+0000","last_unstale":"2024-03-26T21:43:46.810581+0000","last_undegraded":"2024-03-26T21:43:46.810581+0000","last_fullsized":"2024-03-26T21:43:46.810581+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:43.984446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:43.984446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":801,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807406+0000","last_change":"2024-03-26T21:41:25.722685+0000","last_active":"2024-03-26T21:43:46.807406+0000","last_peered":"2024-03-26T21:43:46.807406+0000","last_clean":"2024-03-26T21:43:46.807406+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:43:46.807406+0000","last_undegraded":"2024-03-26T21:43:46.807406+0000","last_fullsized":"2024-03-26T21:43:46.807406+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:39:24.057586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:39:24.057586+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809746+0000","last_change":"2024-03-26T21:41:06.675109+0000","last_active":"2024-03-26T21:43:46.809746+0000","last_peered":"2024-03-26T21:43:46.809746+0000","last_clean":"2024-03-26T21:43:46.809746+0000","last_became_active":"2024-03-26T21:41:06.674713+0000","last_became_peered":"2024-03-26T21:41:06.674713+0000","last_unstale":"2024-03-26T21:43:46.809746+0000","last_undegraded":"2024-03-26T21:43:46.809746+0000","last_fullsized":"2024-03-26T21:43:46.809746+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:40:41.689096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":916,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.883374+0000","last_change":"2024-03-26T21:43:03.102559+0000","last_active":"2024-03-26T21:43:44.883374+0000","last_peered":"2024-03-26T21:43:44.883374+0000","last_clean":"2024-03-26T21:43:44.883374+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:43:44.883374+0000","last_undegraded":"2024-03-26T21:43:44.883374+0000","last_fullsized":"2024-03-26T21:43:44.883374+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:43:03.102463+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:43:03.102463+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:23.082632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":541,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808155+0000","last_change":"2024-03-26T21:41:25.608542+0000","last_active":"2024-03-26T21:43:46.808155+0000","last_peered":"2024-03-26T21:43:46.808155+0000","last_clean":"2024-03-26T21:43:46.808155+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:43:46.808155+0000","last_undegraded":"2024-03-26T21:43:46.808155+0000","last_fullsized":"2024-03-26T21:43:46.808155+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:25.982541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:39:25.982541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"340'1411","reported_seq":1443,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.594384+0000","last_change":"2024-03-26T21:43:28.828681+0000","last_active":"2024-03-26T21:43:48.594384+0000","last_peered":"2024-03-26T21:43:48.594384+0000","last_clean":"2024-03-26T21:43:48.594384+0000","last_became_active":"2024-03-26T21:43:28.828352+0000","last_became_peered":"2024-03-26T21:43:28.828352+0000","last_unstale":"2024-03-26T21:43:48.594384+0000","last_undegraded":"2024-03-26T21:43:48.594384+0000","last_fullsized":"2024-03-26T21:43:48.594384+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1411,"log_dups_size":0,"ondisk_log_size":1411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:32.194151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":4233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1411,"num_write_kb":90304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":614,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802222+0000","last_change":"2024-03-26T21:41:06.698565+0000","last_active":"2024-03-26T21:43:46.802222+0000","last_peered":"2024-03-26T21:43:46.802222+0000","last_clean":"2024-03-26T21:43:46.802222+0000","last_became_active":"2024-03-26T21:41:06.698203+0000","last_became_peered":"2024-03-26T21:41:06.698203+0000","last_unstale":"2024-03-26T21:43:46.802222+0000","last_undegraded":"2024-03-26T21:43:46.802222+0000","last_fullsized":"2024-03-26T21:43:46.802222+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:21.924941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:21.924941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":631,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806095+0000","last_change":"2024-03-26T21:41:06.678442+0000","last_active":"2024-03-26T21:43:46.806095+0000","last_peered":"2024-03-26T21:43:46.806095+0000","last_clean":"2024-03-26T21:43:46.806095+0000","last_became_active":"2024-03-26T21:41:06.676496+0000","last_became_peered":"2024-03-26T21:41:06.676496+0000","last_unstale":"2024-03-26T21:43:46.806095+0000","last_undegraded":"2024-03-26T21:43:46.806095+0000","last_fullsized":"2024-03-26T21:43:46.806095+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:40:47.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1364,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:47.070470+0000","last_change":"2024-03-26T21:41:25.722301+0000","last_active":"2024-03-26T21:43:47.070470+0000","last_peered":"2024-03-26T21:43:47.070470+0000","last_clean":"2024-03-26T21:43:47.070470+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:43:47.070470+0000","last_undegraded":"2024-03-26T21:43:47.070470+0000","last_fullsized":"2024-03-26T21:43:47.070470+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:39:33.064714+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":793,"num_read_kb":788,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":656,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805450+0000","last_change":"2024-03-26T21:40:16.682329+0000","last_active":"2024-03-26T21:43:46.805450+0000","last_peered":"2024-03-26T21:43:46.805450+0000","last_clean":"2024-03-26T21:43:46.805450+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:43:46.805450+0000","last_undegraded":"2024-03-26T21:43:46.805450+0000","last_fullsized":"2024-03-26T21:43:46.805450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:16.682154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.682154+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":603,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.811068+0000","last_change":"2024-03-26T21:40:49.926871+0000","last_active":"2024-03-26T21:43:46.811068+0000","last_peered":"2024-03-26T21:43:46.811068+0000","last_clean":"2024-03-26T21:43:46.811068+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:43:46.811068+0000","last_undegraded":"2024-03-26T21:43:46.811068+0000","last_fullsized":"2024-03-26T21:43:46.811068+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.926754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.926754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"339'1095","reported_seq":1125,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.764034+0000","last_change":"2024-03-26T21:43:28.830803+0000","last_active":"2024-03-26T21:43:46.764034+0000","last_peered":"2024-03-26T21:43:46.764034+0000","last_clean":"2024-03-26T21:43:46.764034+0000","last_became_active":"2024-03-26T21:43:28.829451+0000","last_became_peered":"2024-03-26T21:43:28.829451+0000","last_unstale":"2024-03-26T21:43:46.764034+0000","last_undegraded":"2024-03-26T21:43:46.764034+0000","last_fullsized":"2024-03-26T21:43:46.764034+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:36.380439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1095,"num_write_kb":70080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":600,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801726+0000","last_change":"2024-03-26T21:39:39.919957+0000","last_active":"2024-03-26T21:43:46.801726+0000","last_peered":"2024-03-26T21:43:46.801726+0000","last_clean":"2024-03-26T21:43:46.801726+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:43:46.801726+0000","last_undegraded":"2024-03-26T21:43:46.801726+0000","last_fullsized":"2024-03-26T21:43:46.801726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:39.919791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:39:39.919791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802431+0000","last_change":"2024-03-26T21:40:18.905838+0000","last_active":"2024-03-26T21:43:46.802431+0000","last_peered":"2024-03-26T21:43:46.802431+0000","last_clean":"2024-03-26T21:43:46.802431+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:43:46.802431+0000","last_undegraded":"2024-03-26T21:43:46.802431+0000","last_fullsized":"2024-03-26T21:43:46.802431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:18.905727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:40:18.905727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":660,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806843+0000","last_change":"2024-03-26T21:43:04.127568+0000","last_active":"2024-03-26T21:43:46.806843+0000","last_peered":"2024-03-26T21:43:46.806843+0000","last_clean":"2024-03-26T21:43:46.806843+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:43:46.806843+0000","last_undegraded":"2024-03-26T21:43:46.806843+0000","last_fullsized":"2024-03-26T21:43:46.806843+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:43:04.127429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:04.127429+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.683211+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":610,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.882834+0000","last_change":"2024-03-26T21:41:45.041708+0000","last_active":"2024-03-26T21:43:44.882834+0000","last_peered":"2024-03-26T21:43:44.882834+0000","last_clean":"2024-03-26T21:43:44.882834+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:43:44.882834+0000","last_undegraded":"2024-03-26T21:43:44.882834+0000","last_fullsized":"2024-03-26T21:43:44.882834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:45.041580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:41:45.041580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"340'1305","reported_seq":1337,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.236097+0000","last_change":"2024-03-26T21:43:28.831132+0000","last_active":"2024-03-26T21:43:48.236097+0000","last_peered":"2024-03-26T21:43:48.236097+0000","last_clean":"2024-03-26T21:43:48.236097+0000","last_became_active":"2024-03-26T21:43:28.830782+0000","last_became_peered":"2024-03-26T21:43:28.830782+0000","last_unstale":"2024-03-26T21:43:48.236097+0000","last_undegraded":"2024-03-26T21:43:48.236097+0000","last_fullsized":"2024-03-26T21:43:48.236097+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1305,"log_dups_size":0,"ondisk_log_size":1305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:53.992154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":83520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":598,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.806722+0000","last_change":"2024-03-26T21:41:06.685939+0000","last_active":"2024-03-26T21:43:46.806722+0000","last_peered":"2024-03-26T21:43:46.806722+0000","last_clean":"2024-03-26T21:43:46.806722+0000","last_became_active":"2024-03-26T21:41:06.683042+0000","last_became_peered":"2024-03-26T21:41:06.683042+0000","last_unstale":"2024-03-26T21:43:46.806722+0000","last_undegraded":"2024-03-26T21:43:46.806722+0000","last_fullsized":"2024-03-26T21:43:46.806722+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:28.770387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:28.770387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810177+0000","last_change":"2024-03-26T21:41:06.693074+0000","last_active":"2024-03-26T21:43:46.810177+0000","last_peered":"2024-03-26T21:43:46.810177+0000","last_clean":"2024-03-26T21:43:46.810177+0000","last_became_active":"2024-03-26T21:41:06.692529+0000","last_became_peered":"2024-03-26T21:41:06.692529+0000","last_unstale":"2024-03-26T21:43:46.810177+0000","last_undegraded":"2024-03-26T21:43:46.810177+0000","last_fullsized":"2024-03-26T21:43:46.810177+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:37.945472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:40:37.945472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":683,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802733+0000","last_change":"2024-03-26T21:40:29.968002+0000","last_active":"2024-03-26T21:43:46.802733+0000","last_peered":"2024-03-26T21:43:46.802733+0000","last_clean":"2024-03-26T21:43:46.802733+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:43:46.802733+0000","last_undegraded":"2024-03-26T21:43:46.802733+0000","last_fullsized":"2024-03-26T21:43:46.802733+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:40:29.967581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.967581+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":787,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805169+0000","last_change":"2024-03-26T21:40:27.726654+0000","last_active":"2024-03-26T21:43:46.805169+0000","last_peered":"2024-03-26T21:43:46.805169+0000","last_clean":"2024-03-26T21:43:46.805169+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:43:46.805169+0000","last_undegraded":"2024-03-26T21:43:46.805169+0000","last_fullsized":"2024-03-26T21:43:46.805169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:40:27.726492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:27.726492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"339'1069","reported_seq":1099,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.769362+0000","last_change":"2024-03-26T21:43:28.832520+0000","last_active":"2024-03-26T21:43:46.769362+0000","last_peered":"2024-03-26T21:43:46.769362+0000","last_clean":"2024-03-26T21:43:46.769362+0000","last_became_active":"2024-03-26T21:43:28.832126+0000","last_became_peered":"2024-03-26T21:43:28.832126+0000","last_unstale":"2024-03-26T21:43:46.769362+0000","last_undegraded":"2024-03-26T21:43:46.769362+0000","last_fullsized":"2024-03-26T21:43:46.769362+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:48.627504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70057984,"num_objects":1069,"num_object_clones":0,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1069,"num_write_kb":68416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":524,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807582+0000","last_change":"2024-03-26T21:43:00.377290+0000","last_active":"2024-03-26T21:43:46.807582+0000","last_peered":"2024-03-26T21:43:46.807582+0000","last_clean":"2024-03-26T21:43:46.807582+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:43:46.807582+0000","last_undegraded":"2024-03-26T21:43:46.807582+0000","last_fullsized":"2024-03-26T21:43:46.807582+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.377185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.377185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:16.298207+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":618,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805553+0000","last_change":"2024-03-26T21:39:45.352141+0000","last_active":"2024-03-26T21:43:46.805553+0000","last_peered":"2024-03-26T21:43:46.805553+0000","last_clean":"2024-03-26T21:43:46.805553+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:43:46.805553+0000","last_undegraded":"2024-03-26T21:43:46.805553+0000","last_fullsized":"2024-03-26T21:43:46.805553+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:39:45.352017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.352017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":345,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802817+0000","last_change":"2024-03-26T21:42:52.675227+0000","last_active":"2024-03-26T21:43:46.802817+0000","last_peered":"2024-03-26T21:43:46.802817+0000","last_clean":"2024-03-26T21:43:46.802817+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:43:46.802817+0000","last_undegraded":"2024-03-26T21:43:46.802817+0000","last_fullsized":"2024-03-26T21:43:46.802817+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:42:52.674814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.674814+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:01.007259+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"270'13","reported_seq":215,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.801623+0000","last_change":"2024-03-26T21:41:06.698496+0000","last_active":"2024-03-26T21:43:46.801623+0000","last_peered":"2024-03-26T21:43:46.801623+0000","last_clean":"2024-03-26T21:43:46.801623+0000","last_became_active":"2024-03-26T21:41:06.690006+0000","last_became_peered":"2024-03-26T21:41:06.690006+0000","last_unstale":"2024-03-26T21:43:46.801623+0000","last_undegraded":"2024-03-26T21:43:46.801623+0000","last_fullsized":"2024-03-26T21:43:46.801623+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:40:23.374040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.374040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"340'1382","reported_seq":1414,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.670789+0000","last_change":"2024-03-26T21:43:28.833912+0000","last_active":"2024-03-26T21:43:48.670789+0000","last_peered":"2024-03-26T21:43:48.670789+0000","last_clean":"2024-03-26T21:43:48.670789+0000","last_became_active":"2024-03-26T21:43:28.833363+0000","last_became_peered":"2024-03-26T21:43:28.833363+0000","last_unstale":"2024-03-26T21:43:48.670789+0000","last_undegraded":"2024-03-26T21:43:48.670789+0000","last_fullsized":"2024-03-26T21:43:48.670789+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:41.390730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1382,"num_write_kb":88448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":314,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802778+0000","last_change":"2024-03-26T21:41:43.000293+0000","last_active":"2024-03-26T21:43:46.802778+0000","last_peered":"2024-03-26T21:43:46.802778+0000","last_clean":"2024-03-26T21:43:46.802778+0000","last_became_active":"2024-03-26T21:41:06.682029+0000","last_became_peered":"2024-03-26T21:41:06.682029+0000","last_unstale":"2024-03-26T21:43:46.802778+0000","last_undegraded":"2024-03-26T21:43:46.802778+0000","last_fullsized":"2024-03-26T21:43:46.802778+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:43.000172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:41:43.000172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":146,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.808388+0000","last_change":"2024-03-26T21:43:08.323733+0000","last_active":"2024-03-26T21:43:46.808388+0000","last_peered":"2024-03-26T21:43:46.808388+0000","last_clean":"2024-03-26T21:43:46.808388+0000","last_became_active":"2024-03-26T21:41:26.606051+0000","last_became_peered":"2024-03-26T21:41:26.606051+0000","last_unstale":"2024-03-26T21:43:46.808388+0000","last_undegraded":"2024-03-26T21:43:46.808388+0000","last_fullsized":"2024-03-26T21:43:46.808388+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:08.323649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.323649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:19.730481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"317'38","reported_seq":572,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.807134+0000","last_change":"2024-03-26T21:42:54.349059+0000","last_active":"2024-03-26T21:43:46.807134+0000","last_peered":"2024-03-26T21:43:46.807134+0000","last_clean":"2024-03-26T21:43:46.807134+0000","last_became_active":"2024-03-26T21:41:26.607985+0000","last_became_peered":"2024-03-26T21:41:26.607985+0000","last_unstale":"2024-03-26T21:43:46.807134+0000","last_undegraded":"2024-03-26T21:43:46.807134+0000","last_fullsized":"2024-03-26T21:43:46.807134+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.348885+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:07.702055+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":313,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803022+0000","last_change":"2024-03-26T21:41:28.893317+0000","last_active":"2024-03-26T21:43:46.803022+0000","last_peered":"2024-03-26T21:43:46.803022+0000","last_clean":"2024-03-26T21:43:46.803022+0000","last_became_active":"2024-03-26T21:41:26.630837+0000","last_became_peered":"2024-03-26T21:41:26.630837+0000","last_unstale":"2024-03-26T21:43:46.803022+0000","last_undegraded":"2024-03-26T21:43:46.803022+0000","last_fullsized":"2024-03-26T21:43:46.803022+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:28.893213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:41:28.893213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"340'1394","reported_seq":1426,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:48.664551+0000","last_change":"2024-03-26T21:43:28.829103+0000","last_active":"2024-03-26T21:43:48.664551+0000","last_peered":"2024-03-26T21:43:48.664551+0000","last_clean":"2024-03-26T21:43:48.664551+0000","last_became_active":"2024-03-26T21:43:28.828751+0000","last_became_peered":"2024-03-26T21:43:28.828751+0000","last_unstale":"2024-03-26T21:43:48.664551+0000","last_undegraded":"2024-03-26T21:43:48.664551+0000","last_fullsized":"2024-03-26T21:43:48.664551+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1394,"log_dups_size":0,"ondisk_log_size":1394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:47.189407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91357184,"num_objects":1394,"num_object_clones":0,"num_object_copies":4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1394,"num_write_kb":89216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":598,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809622+0000","last_change":"2024-03-26T21:43:01.304736+0000","last_active":"2024-03-26T21:43:46.809622+0000","last_peered":"2024-03-26T21:43:46.809622+0000","last_clean":"2024-03-26T21:43:46.809622+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:43:46.809622+0000","last_undegraded":"2024-03-26T21:43:46.809622+0000","last_fullsized":"2024-03-26T21:43:46.809622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.304625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.304625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:09.389262+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":396,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803382+0000","last_change":"2024-03-26T21:41:24.943122+0000","last_active":"2024-03-26T21:43:46.803382+0000","last_peered":"2024-03-26T21:43:46.803382+0000","last_clean":"2024-03-26T21:43:46.803382+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:43:46.803382+0000","last_undegraded":"2024-03-26T21:43:46.803382+0000","last_fullsized":"2024-03-26T21:43:46.803382+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:41:24.943030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:41:24.943030+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":576,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802045+0000","last_change":"2024-03-26T21:43:06.961745+0000","last_active":"2024-03-26T21:43:46.802045+0000","last_peered":"2024-03-26T21:43:46.802045+0000","last_clean":"2024-03-26T21:43:46.802045+0000","last_became_active":"2024-03-26T21:41:26.608779+0000","last_became_peered":"2024-03-26T21:41:26.608779+0000","last_unstale":"2024-03-26T21:43:46.802045+0000","last_undegraded":"2024-03-26T21:43:46.802045+0000","last_fullsized":"2024-03-26T21:43:46.802045+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.961290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:36.730992+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":729,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802975+0000","last_change":"2024-03-26T21:41:26.670298+0000","last_active":"2024-03-26T21:43:46.802975+0000","last_peered":"2024-03-26T21:43:46.802975+0000","last_clean":"2024-03-26T21:43:46.802975+0000","last_became_active":"2024-03-26T21:41:26.613401+0000","last_became_peered":"2024-03-26T21:41:26.613401+0000","last_unstale":"2024-03-26T21:43:46.802975+0000","last_undegraded":"2024-03-26T21:43:46.802975+0000","last_fullsized":"2024-03-26T21:43:46.802975+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:14.873121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:14.873121+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"339'1077","reported_seq":1107,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.769744+0000","last_change":"2024-03-26T21:43:28.833360+0000","last_active":"2024-03-26T21:43:46.769744+0000","last_peered":"2024-03-26T21:43:46.769744+0000","last_clean":"2024-03-26T21:43:46.769744+0000","last_became_active":"2024-03-26T21:43:28.833018+0000","last_became_peered":"2024-03-26T21:43:28.833018+0000","last_unstale":"2024-03-26T21:43:46.769744+0000","last_undegraded":"2024-03-26T21:43:46.769744+0000","last_fullsized":"2024-03-26T21:43:46.769744+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:43.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1077,"num_write_kb":68928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":164,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.881950+0000","last_change":"2024-03-26T21:42:53.182450+0000","last_active":"2024-03-26T21:43:44.881950+0000","last_peered":"2024-03-26T21:43:44.881950+0000","last_clean":"2024-03-26T21:43:44.881950+0000","last_became_active":"2024-03-26T21:41:26.594420+0000","last_became_peered":"2024-03-26T21:41:26.594420+0000","last_unstale":"2024-03-26T21:43:44.881950+0000","last_undegraded":"2024-03-26T21:43:44.881950+0000","last_fullsized":"2024-03-26T21:43:44.881950+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:53.182329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:42:53.182329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:43:54.585869+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":615,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810949+0000","last_change":"2024-03-26T21:43:05.204186+0000","last_active":"2024-03-26T21:43:46.810949+0000","last_peered":"2024-03-26T21:43:46.810949+0000","last_clean":"2024-03-26T21:43:46.810949+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:43:46.810949+0000","last_undegraded":"2024-03-26T21:43:46.810949+0000","last_fullsized":"2024-03-26T21:43:46.810949+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.204079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.204079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:06.043660+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":309,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802715+0000","last_change":"2024-03-26T21:41:33.902737+0000","last_active":"2024-03-26T21:43:46.802715+0000","last_peered":"2024-03-26T21:43:46.802715+0000","last_clean":"2024-03-26T21:43:46.802715+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:43:46.802715+0000","last_undegraded":"2024-03-26T21:43:46.802715+0000","last_fullsized":"2024-03-26T21:43:46.802715+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:33.902465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:33.902465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":332,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803296+0000","last_change":"2024-03-26T21:42:57.692467+0000","last_active":"2024-03-26T21:43:46.803296+0000","last_peered":"2024-03-26T21:43:46.803296+0000","last_clean":"2024-03-26T21:43:46.803296+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:43:46.803296+0000","last_undegraded":"2024-03-26T21:43:46.803296+0000","last_fullsized":"2024-03-26T21:43:46.803296+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:57.692111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.692111+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:02.432141+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"340'1134","reported_seq":1166,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.933422+0000","last_change":"2024-03-26T21:43:28.826710+0000","last_active":"2024-03-26T21:43:46.933422+0000","last_peered":"2024-03-26T21:43:46.933422+0000","last_clean":"2024-03-26T21:43:46.933422+0000","last_became_active":"2024-03-26T21:43:28.826397+0000","last_became_peered":"2024-03-26T21:43:28.826397+0000","last_unstale":"2024-03-26T21:43:46.933422+0000","last_undegraded":"2024-03-26T21:43:46.933422+0000","last_fullsized":"2024-03-26T21:43:46.933422+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:36.055328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1134,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":634,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.810538+0000","last_change":"2024-03-26T21:42:58.401711+0000","last_active":"2024-03-26T21:43:46.810538+0000","last_peered":"2024-03-26T21:43:46.810538+0000","last_clean":"2024-03-26T21:43:46.810538+0000","last_became_active":"2024-03-26T21:41:06.685057+0000","last_became_peered":"2024-03-26T21:41:06.685057+0000","last_unstale":"2024-03-26T21:43:46.810538+0000","last_undegraded":"2024-03-26T21:43:46.810538+0000","last_fullsized":"2024-03-26T21:43:46.810538+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.401600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.401600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:21.544808+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":633,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802550+0000","last_change":"2024-03-26T21:40:22.891092+0000","last_active":"2024-03-26T21:43:46.802550+0000","last_peered":"2024-03-26T21:43:46.802550+0000","last_clean":"2024-03-26T21:43:46.802550+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:43:46.802550+0000","last_undegraded":"2024-03-26T21:43:46.802550+0000","last_fullsized":"2024-03-26T21:43:46.802550+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.890287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.890287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":639,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-26T21:43:44.882397+0000","last_change":"2024-03-26T21:41:38.113058+0000","last_active":"2024-03-26T21:43:44.882397+0000","last_peered":"2024-03-26T21:43:44.882397+0000","last_clean":"2024-03-26T21:43:44.882397+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:43:44.882397+0000","last_undegraded":"2024-03-26T21:43:44.882397+0000","last_fullsized":"2024-03-26T21:43:44.882397+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.112933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.112933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":355,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.802413+0000","last_change":"2024-03-26T21:41:50.859131+0000","last_active":"2024-03-26T21:43:46.802413+0000","last_peered":"2024-03-26T21:43:46.802413+0000","last_clean":"2024-03-26T21:43:46.802413+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:43:46.802413+0000","last_undegraded":"2024-03-26T21:43:46.802413+0000","last_fullsized":"2024-03-26T21:43:46.802413+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:41:50.858942+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"340'1228","reported_seq":1260,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:47.655274+0000","last_change":"2024-03-26T21:43:28.831488+0000","last_active":"2024-03-26T21:43:47.655274+0000","last_peered":"2024-03-26T21:43:47.655274+0000","last_clean":"2024-03-26T21:43:47.655274+0000","last_became_active":"2024-03-26T21:43:28.831192+0000","last_became_peered":"2024-03-26T21:43:28.831192+0000","last_unstale":"2024-03-26T21:43:47.655274+0000","last_undegraded":"2024-03-26T21:43:47.655274+0000","last_fullsized":"2024-03-26T21:43:47.655274+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":1228,"log_dups_size":0,"ondisk_log_size":1228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:49.971151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80478208,"num_objects":1228,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1228,"num_write_kb":78592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":308,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803271+0000","last_change":"2024-03-26T21:41:56.818280+0000","last_active":"2024-03-26T21:43:46.803271+0000","last_peered":"2024-03-26T21:43:46.803271+0000","last_clean":"2024-03-26T21:43:46.803271+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:43:46.803271+0000","last_undegraded":"2024-03-26T21:43:46.803271+0000","last_fullsized":"2024-03-26T21:43:46.803271+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:41:56.817883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":814,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.809840+0000","last_change":"2024-03-26T21:42:45.315356+0000","last_active":"2024-03-26T21:43:46.809840+0000","last_peered":"2024-03-26T21:43:46.809840+0000","last_clean":"2024-03-26T21:43:46.809840+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:43:46.809840+0000","last_undegraded":"2024-03-26T21:43:46.809840+0000","last_fullsized":"2024-03-26T21:43:46.809840+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:42:45.315022+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:42:45.315022+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:05.152774+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":463,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.805421+0000","last_change":"2024-03-26T21:41:06.677479+0000","last_active":"2024-03-26T21:43:46.805421+0000","last_peered":"2024-03-26T21:43:46.805421+0000","last_clean":"2024-03-26T21:43:46.805421+0000","last_became_active":"2024-03-26T21:41:06.675316+0000","last_became_peered":"2024-03-26T21:41:06.675316+0000","last_unstale":"2024-03-26T21:43:46.805421+0000","last_undegraded":"2024-03-26T21:43:46.805421+0000","last_fullsized":"2024-03-26T21:43:46.805421+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.653184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.653184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":371,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-26T21:43:46.803078+0000","last_change":"2024-03-26T21:43:16.802920+0000","last_active":"2024-03-26T21:43:46.803078+0000","last_peered":"2024-03-26T21:43:46.803078+0000","last_clean":"2024-03-26T21:43:46.803078+0000","last_became_active":"2024-03-26T21:43:16.796404+0000","last_became_peered":"2024-03-26T21:43:16.796404+0000","last_unstale":"2024-03-26T21:43:46.803078+0000","last_undegraded":"2024-03-26T21:43:46.803078+0000","last_fullsized":"2024-03-26T21:43:46.803078+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.600649+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:44:15.962250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1251475456,"num_objects":19096,"num_object_clones":0,"num_object_copies":57288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19096,"num_write_kb":1222144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3961126912,"data_stored":3961126912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19096,"ondisk_log_size":19096,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":793,"num_read_kb":788,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":4544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":11766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973726,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690600,"kb_used_data":420508,"kb_used_omap":6,"kb_used_meta":270073,"kb_avail":93054552,"statfs":{"total":95995035648,"available":95287861248,"internally_reserved":0,"allocated":430600192,"data_stored":429952682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":276555048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364993,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831648,"kb_used_data":471444,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92913504,"statfs":{"total":95995035648,"available":95143428096,"internally_reserved":0,"allocated":482758656,"data_stored":482110830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839928,"kb_used_data":299244,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92905224,"statfs":{"total":95995035648,"available":95134949376,"internally_reserved":0,"allocated":306425856,"data_stored":305760869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1426728,"kb_used_data":840224,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92318424,"statfs":{"total":95995035648,"available":94534066176,"internally_reserved":0,"allocated":860389376,"data_stored":859737186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411541,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928700,"kb_used_data":456048,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92816452,"statfs":{"total":95995035648,"available":95044046848,"internally_reserved":0,"allocated":466993152,"data_stored":466381137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149903,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650500,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309864,"kb_used_data":866780,"kb_used_omap":4,"kb_used_meta":443067,"kb_avail":92435288,"statfs":{"total":95995035648,"available":94653734912,"internally_reserved":0,"allocated":887582720,"data_stored":886934109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":453701283},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659332,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":857228,"kb_used_data":324868,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92887924,"statfs":{"total":95995035648,"available":95117234176,"internally_reserved":0,"allocated":332664832,"data_stored":332032796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":913178624,"data_stored":913178624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324861952,"data_stored":324861952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":509411328,"data_stored":509411328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":906821632,"data_stored":906821632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":370933760,"data_stored":370933760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":502530048,"data_stored":502530048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":433389568,"data_stored":433389568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:43:53.665 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T21:43:53.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-26T21:43:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:53.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:53.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:53.791 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:53.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:53.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:53.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:54.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:54.066 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:54.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:54.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:54.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:43:54.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:43:54.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:43:54.251 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:43:54.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:43:54.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:43:54.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:43:54.252 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:54 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:43:54.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 13 16 33648 33632 161.648 90.1875 0.00426913 0.00613739 2024-03-26T21:43:54.293 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:54.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:54.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:54 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:43:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:54 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:43:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:54 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:43:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:54 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:43:54.545 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:54.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:54.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:54.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:54.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:54.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:54.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:55.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:55.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:55.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:55.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 14 16 35513 35497 158.428 116.562 0.00495564 0.00630593 2024-03-26T21:43:55.345 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:43:55.385 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:43:55.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:43:55.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[40587]: pgmap v700: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T21:43:55.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:43:55.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:43:55.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:43:55.479 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:43:55.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[29406]: pgmap v700: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T21:43:55.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:43:55.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:43:55.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:43:55.480 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:55 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:43:55.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:55.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:55 smithi067 ceph-mon[33346]: pgmap v700: 148 pgs: 148 active+clean; 1.7 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 170 MiB/s wr, 2.71k op/s 2024-03-26T21:43:55.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:55 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:43:55.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:55 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:43:55.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:55 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:43:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:55 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:43:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:55.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:55.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:55.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:55.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:55.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:55.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:55.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:56.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:56.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:56.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:56.220 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:56.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 15 16 37470 37454 156.02 122.312 0.00499457 0.00640415 2024-03-26T21:43:56.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:56 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:43:56.308 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:56 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:43:56.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:56 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:43:56.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:56 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:43:56.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:56.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:56.397 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:56.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:56.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:56 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:43:56.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:56 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:43:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:56.562 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:43:56.602 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:56.648 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:56.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:56.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:56.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:56.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:56.989 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:43:57.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:57.062 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:57.164 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:57.205 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:57.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 16 16 39431 39415 153.929 122.562 0.00744978 0.0064912 2024-03-26T21:43:57.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:57.342 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:57.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: pgmap v701: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.63k op/s 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 6.6 scrub starts 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 6.6 scrub ok 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:43:57.564 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: pgmap v701: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.63k op/s 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 6.6 scrub starts 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 6.6 scrub ok 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:43:57.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:57 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:43:57.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:43:57.615 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:43:57.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:57.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: pgmap v701: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.63k op/s 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:43:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:43:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 6.6 scrub starts 2024-03-26T21:43:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 6.6 scrub ok 2024-03-26T21:43:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:43:57.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:57 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:43:57.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:57.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:57.894 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:57.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:57.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:58.031 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:58.133 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:58.169 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:58.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 17 16 41405 41389 152.131 123.375 0.00492821 0.00656726 2024-03-26T21:43:58.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:43:58.307 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:43:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:43:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:43:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:43:58.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:58.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:58.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:58.585 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:58.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:58.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:43:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:58 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:43:58.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:58.865 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:58.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:58.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:59.005 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:59.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:43:59.140 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:43:59.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:43:59.253 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:43:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:43:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:43:59.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 18 16 43326 43310 150.349 120.062 0.00568882 0.00664621 2024-03-26T21:43:59.278 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:43:59.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:43:59.418 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:43:59.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:43:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:43:59.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:43:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[40587]: pgmap v702: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 946 B/s rd, 152 MiB/s wr, 2.43k op/s 2024-03-26T21:43:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:43:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:43:59.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[29406]: pgmap v702: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 946 B/s rd, 152 MiB/s wr, 2.43k op/s 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:43:59.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:43:59 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:43:59.660 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.698 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:59 smithi067 ceph-mon[33346]: pgmap v702: 148 pgs: 148 active+clean; 2.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 946 B/s rd, 152 MiB/s wr, 2.43k op/s 2024-03-26T21:43:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:59 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:43:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:59 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:43:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:59 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:43:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:43:59 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:43:59.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:43:59.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:43:59.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:43:59.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:43:59.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:00.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:00.135 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:00.157 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:00.256 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 19 16 45139 45123 148.397 113.312 0.00733401 0.00673257 2024-03-26T21:44:00.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:00.271 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:44:00.271 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:44:00.271 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:44:00.271 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:44:00.307 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:00.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:00.446 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:00.522 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:44:00.522 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:44:00.522 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:44:00.522 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:00 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:44:00.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:00.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:00.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:00 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:44:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:00 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:44:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:00 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:44:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:00 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:44:00.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:00.862 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:00.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:00.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:01.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:01.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:01.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:44:01.265158+0000 min lat: 0.00140536 max lat: 0.169332 avg lat: 0.00681588 2024-03-26T21:44:01.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:44:01.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 20 16 46933 46917 146.583 112.125 0.00944684 0.00681588 2024-03-26T21:44:01.277 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:01.287 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:44:01.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:01.402 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:01.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:01.540 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:01.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: pgmap v703: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3964412718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:44:01.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: pgmap v703: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3964412718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:44:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:01 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:44:01.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:01.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: pgmap v703: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3964412718' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:44:01.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:44:01.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:44:01.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:44:01.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:01 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:44:01.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:01.826 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:01.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:01.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:01.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:02.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:02.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:02.241 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:02.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 21 16 48566 48550 144.463 102.062 0.00985439 0.0069173 2024-03-26T21:44:02.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:02.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:02.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:02.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: osdmap e341: 8 total, 8 up, 7 in 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: osdmap e342: 8 total, 8 up, 7 in 2024-03-26T21:44:02.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: osdmap e341: 8 total, 8 up, 7 in 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: osdmap e342: 8 total, 8 up, 7 in 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:02.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:02.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.655 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:02.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: osdmap e341: 8 total, 8 up, 7 in 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: osdmap e342: 8 total, 8 up, 7 in 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T21:44:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:02.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:02.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:02.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:02.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:02.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:02.973 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147838 9.4 9:21976012:::benchmark_data_smithi055_135863_object17187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.657182+0000", 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.319262874, 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2044302459999998, 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:02.974 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147838 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657182+0000", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657182+0000", 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.975 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657183+0000", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0160000000000001e-06 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657188+0000", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0429999999999998e-06 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657199+0000", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0129000000000001e-05 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:02.976 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.657205+0000", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1999999999999999e-06 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.661386+0000", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0041815350000000001 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.661529+0000", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014289199999999999 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.661663+0000", 2024-03-26T21:44:02.977 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013359 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.667466+0000", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0058033820000000002 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.706724+0000", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.039258154000000003 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.706755+0000", 2024-03-26T21:44:02.978 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0411000000000001e-05 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.674099+0000", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96734383800000001 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.674115+0000", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6041000000000001e-05 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.840395+0000", 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1662808600000001 2024-03-26T21:44:02.979 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for missing object", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.840516+0000", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120735 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.861509+0000", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020992647 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.861613+0000", 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103773 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:02.980 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:2866 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94208173342592 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.658266+0000", 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.318179195, 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0603410639999999, 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 2866 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:02.981 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658266+0000", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658266+0000", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658267+0000", 2024-03-26T21:44:02.982 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0809999999999999e-06 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658273+0000", 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5790000000000003e-06 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658290+0000", 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7462999999999999e-05 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:02.983 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658301+0000", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0767e-05 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.700329+0000", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.042027745999999998 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.700341+0000", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2072999999999999e-05 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.690967+0000", 2024-03-26T21:44:02.984 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99062587700000004 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for degraded object", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.691371+0000", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00040464800000000002 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.697172+0000", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.005800464 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.697265+0000", 2024-03-26T21:44:02.985 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3210999999999999e-05 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.718607+0000", 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.021342155000000002 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147830 9.2 9:40ad807a:::benchmark_data_smithi055_135863_object17179:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.668091+0000", 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.30835469499999, 2024-03-26T21:44:02.986 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.061567739, 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147830 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668091+0000", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:02.987 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668091+0000", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668091+0000", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0200000000000002e-07 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668095+0000", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8099999999999999e-06 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:02.988 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668103+0000", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7139999999999993e-06 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668109+0000", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6069999999999996e-06 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671008+0000", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028988479999999999 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671018+0000", 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9569999999999997e-06 2024-03-26T21:44:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681623+0000", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.010605142 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681635+0000", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2266000000000001e-05 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.729570+0000", 2024-03-26T21:44:02.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.047935032 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.729658+0000", 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7861000000000005e-05 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147837 9.3 9:c0efbab2:::benchmark_data_smithi055_135863_object17186:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.668120+0000", 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.30832504, 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1213170830000001, 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:02.991 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147837 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668120+0000", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668120+0000", 2024-03-26T21:44:02.992 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668121+0000", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1600000000000002e-07 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668123+0000", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.723e-06 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668129+0000", 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0290000000000002e-06 2024-03-26T21:44:02.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668135+0000", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8590000000000001e-06 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671256+0000", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0031204969999999999 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671263+0000", 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1930000000000004e-06 2024-03-26T21:44:02.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.682618+0000", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011355225 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.682629+0000", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0712e-05 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.675902+0000", 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99327289500000004 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.675911+0000", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9230000000000008e-06 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.787683+0000", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1117723230000001 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.789437+0000", 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0017542879999999999 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:02.996 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147841 9.2 9:43ef9daf:::benchmark_data_smithi055_135863_object17190:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.668144+0000", 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.30830129, 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0909529720000002, 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147841 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:02.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668144+0000", 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668144+0000", 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:02.998 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668144+0000", 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7399999999999999e-07 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668153+0000", 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8109999999999992e-06 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:02.999 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668159+0000", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2949999999999999e-06 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668165+0000", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6030000000000004e-06 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671044+0000", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028789390000000001 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671051+0000", 2024-03-26T21:44:03.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1620000000000002e-06 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681662+0000", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0106104149999999 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681669+0000", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5889999999999997e-06 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.730501+0000", 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0488316200000001 2024-03-26T21:44:03.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for missing object", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.730591+0000", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0617000000000003e-05 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.758717+0000", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.028126011999999999 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.759097+0000", 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00037953500000000002 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.002 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147847 9.3 9:cef64f9a:::benchmark_data_smithi055_135863_object17196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.668743+0000", 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.30770231400001, 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1333468720000002, 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147847 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.003 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668743+0000", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668743+0000", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668744+0000", 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7300000000000005e-07 2024-03-26T21:44:03.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668748+0000", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7740000000000004e-06 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668758+0000", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9729999999999992e-06 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.668766+0000", 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7179999999999996e-06 2024-03-26T21:44:03.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671590+0000", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028240000000000001 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.671598+0000", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4080000000000001e-06 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.682655+0000", 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011057205000000001 2024-03-26T21:44:03.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.682661+0000", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8170000000000004e-06 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.676167+0000", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99350687199999999 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.676176+0000", 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1710000000000002e-06 2024-03-26T21:44:03.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.789844+0000", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1136680750000001 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.790531+0000", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00068704599999999997 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.790871+0000", 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000339754 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.801797+0000", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010926636 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.801841+0000", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4020000000000002e-05 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.801904+0000", 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2990000000000005e-05 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.801972+0000", 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8262000000000005e-05 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.802090+0000", 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117378 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147850 9.2 9:4a12beb8:::benchmark_data_smithi055_135863_object17199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:03.010 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.692831+0000", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.28361456100001, 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0541222399999999, 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147850 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692831+0000", 2024-03-26T21:44:03.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692831+0000", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692832+0000", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.638e-06 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692839+0000", 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2140000000000001e-06 2024-03-26T21:44:03.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692849+0000", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0757999999999999e-05 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692859+0000", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7520000000000008e-06 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.696945+0000", 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004086162 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.696958+0000", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2882e-05 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681694+0000", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.984736155 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.681703+0000", 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4460000000000001e-06 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.730623+0000", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.048919991 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.730743+0000", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012014 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.731207+0000", 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00046397499999999999 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.744051+0000", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012843736999999999 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.744099+0000", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8233999999999999e-05 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.746437+0000", 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0023382580000000002 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.016 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.746553+0000", 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011564 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.746953+0000", 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00040025800000000002 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147862 9.4 9:26cc9c62:::benchmark_data_smithi055_135863_object17211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:44:03.017 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.721847+0000", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 177.25459849200001, 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1454326140000002, 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147862 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721847+0000", 2024-03-26T21:44:03.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721847+0000", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721848+0000", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5370000000000001e-06 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721854+0000", 2024-03-26T21:44:03.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.173e-06 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721869+0000", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4443999999999999e-05 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.721909+0000", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0574999999999997e-05 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.722052+0000", 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000142466 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.722065+0000", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2554e-05 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.675068+0000", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.953003088 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.675078+0000", 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0455000000000001e-05 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.840817+0000", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.16573941 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.840967+0000", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000149586 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.841373+0000", 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00040555099999999998 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.860874+0000", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019501109999999999 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.861005+0000", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013086599999999999 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.861723+0000", 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000717966 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.866990+0000", 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0052674260000000004 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.867058+0000", 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7811000000000005e-05 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.867279+0000", 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00022159600000000001 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166748 9.2 9:44b99035:::benchmark_data_smithi055_135863_object36097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.572279+0000", 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 158.40416643200001, 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1820435490000001, 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.025 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166748 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572279+0000", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572279+0000", 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572280+0000", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.78e-07 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572284+0000", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0389999999999998e-06 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572292+0000", 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0609999999999996e-06 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.572300+0000", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8250000000000005e-06 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585559+0000", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.013259051000000001 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585675+0000", 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000116252 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.028 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585792+0000", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117148 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.592425+0000", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0066324749999999997 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.612198+0000", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.019773656000000001 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.612216+0000", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7538999999999999e-05 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.745900+0000", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1336843970000001 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for missing object", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.746323+0000", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00042261499999999999 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.753903+0000", 2024-03-26T21:44:03.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0075804970000000003 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754322+0000", 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00041901599999999997 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166770 9.3 9:c219b999:::benchmark_data_smithi055_135863_object36119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.601826+0000", 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 158.37461925299999, 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.153013834, 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.031 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166770 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601826+0000", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601826+0000", 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.032 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601827+0000", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2929999999999999e-06 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601833+0000", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1410000000000002e-06 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601845+0000", 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.119e-05 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.033 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.601854+0000", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7550000000000006e-06 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.614384+0000", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012529957 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.614395+0000", 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0709e-05 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.034 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.750016+0000", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1356212450000001 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.750165+0000", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000149049 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.750299+0000", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013363399999999999 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.753914+0000", 2024-03-26T21:44:03.035 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003614716 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754062+0000", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000148436 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754368+0000", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030579400000000002 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754666+0000", 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029852399999999999 2024-03-26T21:44:03.036 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754726+0000", 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9212000000000002e-05 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.754840+0000", 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011417900000000001 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166771 9.2 9:475bc12c:::benchmark_data_smithi055_135863_object36120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:44:03.037 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.602180+0000", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 158.37426544499999, 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.151016936, 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166771 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602180+0000", 2024-03-26T21:44:03.038 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602180+0000", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602180+0000", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7999999999999995e-07 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602185+0000", 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4379999999999997e-06 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.039 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602195+0000", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.029e-05 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.602203+0000", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3429999999999998e-06 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.612293+0000", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010090116999999999 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.040 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.612303+0000", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0536e-05 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.746720+0000", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.134416667 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.746870+0000", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014963500000000001 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 2,3,7", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.747026+0000", 2024-03-26T21:44:03.041 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015669499999999999 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.750128+0000", 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003102053 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.750667+0000", 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00053881199999999995 2024-03-26T21:44:03.042 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752446+0000", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001778628 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752910+0000", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00046410599999999999 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.753058+0000", 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014859399999999999 2024-03-26T21:44:03.043 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.753197+0000", 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013834200000000001 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166762 9.d 9:b5ae9f71:::benchmark_data_smithi055_135863_object36111:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.599944+0000", 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.37650122299999, 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1263926980000001, 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.044 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166762 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.599944+0000", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.599944+0000", 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.045 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.599946+0000", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5510000000000001e-06 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.599953+0000", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0709999999999997e-06 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600245+0000", 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029234399999999999 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.046 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600256+0000", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0597999999999999e-05 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600351+0000", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5466999999999999e-05 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600362+0000", 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1178999999999999e-05 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.047 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.726241+0000", 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1258784799999999 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.726337+0000", 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6008000000000006e-05 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.048 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166779 9.e 9:7df3f252:::benchmark_data_smithi055_135863_object36128:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.600278+0000", 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.37616689199999, 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1791619739999999, 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.049 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166779 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600278+0000", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.050 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600278+0000", 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600279+0000", 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8999999999999996e-07 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600283+0000", 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.194e-06 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.051 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600291+0000", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0050000000000004e-06 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600308+0000", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6342e-05 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600415+0000", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010723899999999999 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.052 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600427+0000", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2305e-05 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.744285+0000", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1438581400000001 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746361+0000", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002075763 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:44:03.053 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.747091+0000", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00073030100000000004 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.765784+0000", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.018692507000000001 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778193+0000", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012408881 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778542+0000", 2024-03-26T21:44:03.054 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000348833 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778592+0000", 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0682e-05 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.779440+0000", 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00084809200000000005 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.055 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166781 9.d 9:bdd99838:::benchmark_data_smithi055_135863_object36130:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.600327+0000", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.376118142, 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1315928980000001, 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166781 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600327+0000", 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.056 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600327+0000", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600328+0000", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7199999999999999e-07 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600331+0000", 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.8949999999999998e-06 2024-03-26T21:44:03.057 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600338+0000", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0720000000000002e-06 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600344+0000", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0839999999999997e-06 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600410+0000", 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6215000000000002e-05 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.058 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.600421+0000", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1443e-05 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.726517+0000", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.126096062 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.726632+0000", 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000114207 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.059 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.726764+0000", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013232000000000001 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.731258+0000", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0044941820000000002 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.731333+0000", 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5073000000000004e-05 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.060 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.731531+0000", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019778600000000001 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.731780+0000", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00024918 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.731920+0000", 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013980700000000001 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.061 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166794 9.e 9:7cf906d5:::benchmark_data_smithi055_135863_object36143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.749701+0000", 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.226744263, 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.030983835, 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166794 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.062 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749701+0000", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749701+0000", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749702+0000", 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4699999999999999e-06 2024-03-26T21:44:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749710+0000", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5519999999999999e-06 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749722+0000", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.201e-05 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749730+0000", 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4239999999999993e-06 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.064 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749809+0000", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8262000000000004e-05 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.749823+0000", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4355999999999999e-05 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.765814+0000", 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.015991342 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.065 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.765942+0000", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012722800000000001 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.766074+0000", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000132123 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778244+0000", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0121706 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.066 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778292+0000", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7681e-05 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.780506+0000", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0022143699999999998 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.780590+0000", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3653999999999996e-05 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.780685+0000", 2024-03-26T21:44:03.067 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4763000000000001e-05 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166805 9.e 9:719de2e6:::benchmark_data_smithi055_135863_object36154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.769378+0000", 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.20706760300001, 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.011857732, 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.068 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166805 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769378+0000", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769378+0000", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769379+0000", 2024-03-26T21:44:03.069 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.536e-06 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769386+0000", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.277e-06 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769399+0000", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2994e-05 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769410+0000", 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0878e-05 2024-03-26T21:44:03.070 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769484+0000", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3823000000000001e-05 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769498+0000", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3862999999999999e-05 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.769121+0000", 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99962275300000003 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.071 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.769247+0000", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000126265 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.769385+0000", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000138431 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778338+0000", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0089522040000000001 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.778389+0000", 2024-03-26T21:44:03.072 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1743000000000002e-05 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.780873+0000", 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024838529999999998 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.781137+0000", 2024-03-26T21:44:03.073 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00026351800000000002 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 10 pg_num to 26 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.781235+0000", 2024-03-26T21:44:03.074 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8593999999999998e-05 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166811 9.d 9:b9ab87cb:::benchmark_data_smithi055_135863_object36160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.781106+0000", 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 157.195339281, 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98600526799999999, 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.075 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166811 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781106+0000", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781106+0000", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781107+0000", 2024-03-26T21:44:03.076 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.096e-06 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781112+0000", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8359999999999999e-06 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781121+0000", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1689999999999998e-06 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781129+0000", 2024-03-26T21:44:03.077 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.354e-06 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781217+0000", 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7315e-05 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.781225+0000", 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5599999999999994e-06 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.078 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.732696+0000", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.95147080299999998 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.732808+0000", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112137 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.732933+0000", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012469 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.079 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736355+0000", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034224619999999998 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736574+0000", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021832899999999999 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.766966+0000", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.030392339000000001 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.767016+0000", 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9483e-05 2024-03-26T21:44:03.080 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.767111+0000", 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5694999999999998e-05 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:214649 9.4 9:22ad094c:::benchmark_data_smithi055_135863_object83998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:17.836166+0000", 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 105.14027903100001, 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99053085699999999, 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.081 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 214649 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836166+0000", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836166+0000", 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.082 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836167+0000", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.114e-06 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836173+0000", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.4389999999999999e-06 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836183+0000", 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0086e-05 2024-03-26T21:44:03.083 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836190+0000", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.153e-06 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836262+0000", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1503999999999998e-05 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.836270+0000", 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7220000000000005e-06 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.084 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.821604+0000", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98533346399999999 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.821696+0000", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2374999999999996e-05 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,4", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.821815+0000", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000118484 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.085 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826570+0000", 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0047550070000000003 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826587+0000", 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6942000000000001e-05 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826627+0000", 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0933000000000003e-05 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.086 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826663+0000", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5598e-05 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826697+0000", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4035999999999997e-05 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:214675 9.4 9:20307154:::benchmark_data_smithi055_135863_object84024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:17.858502+0000", 2024-03-26T21:44:03.087 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 105.11794322199999, 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97300676399999997, 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 214675 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858502+0000", 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.088 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858502+0000", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858503+0000", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.136e-06 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858509+0000", 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.699e-06 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.089 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858522+0000", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2926e-05 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858530+0000", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7610000000000007e-06 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858589+0000", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9267e-05 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.090 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.858598+0000", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7919999999999998e-06 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.824322+0000", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96572438299999996 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.824453+0000", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000130884 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,4", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.824590+0000", 2024-03-26T21:44:03.091 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013760399999999999 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.826770+0000", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002179636 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.831347+0000", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0045768370000000003 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.831396+0000", 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8599e-05 2024-03-26T21:44:03.092 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.831454+0000", 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8048999999999999e-05 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.831509+0000", 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5191000000000001e-05 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:214686 9.4 9:278e468b:::benchmark_data_smithi055_135863_object84035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-03-26T21:44:03.093 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:17.865538+0000", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 105.110907337, 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98137119100000003, 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 214686 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865538+0000", 2024-03-26T21:44:03.094 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865538+0000", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865539+0000", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3799999999999997e-07 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865542+0000", 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6519999999999999e-06 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.095 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865550+0000", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.413e-06 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865555+0000", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1080000000000001e-06 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865684+0000", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000128875 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:44:03.096 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:17.865698+0000", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3963e-05 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.829001+0000", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96330305599999999 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.829122+0000", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121487 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,4", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.829269+0000", 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014676899999999999 2024-03-26T21:44:03.097 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.831608+0000", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002339275 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.836746+0000", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0051379260000000001 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.846814+0000", 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010068257000000001 2024-03-26T21:44:03.098 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.846863+0000", 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8698e-05 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:18.846909+0000", 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6074e-05 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:44:03.099 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:44:03.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:03.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:03.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:03.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 22 16 50256 50240 142.697 105.625 0.0104522 0.00700247 2024-03-26T21:44:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[40587]: pgmap v705: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T21:44:03.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[29406]: pgmap v705: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T21:44:03.350 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:03.381 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:03.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:03.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:03.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:03.655 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:03.672 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T21:44:03.673 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[40587]: osdmap e343: 8 total, 8 up, 7 in 2024-03-26T21:44:03.673 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T21:44:03.673 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:03 smithi055 ceph-mon[29406]: osdmap e343: 8 total, 8 up, 7 in 2024-03-26T21:44:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:03 smithi067 ceph-mon[33346]: pgmap v705: 148 pgs: 148 active+clean; 2.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T21:44:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:03 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1271416440' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-26T21:44:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:03 smithi067 ceph-mon[33346]: osdmap e343: 8 total, 8 up, 7 in 2024-03-26T21:44:03.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:03.794 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:03.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:03.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:04.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:04.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:04.211 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:04.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 23 16 51997 51981 141.224 108.812 0.00724302 0.00707523 2024-03-26T21:44:04.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:04.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:44:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:04.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:04.508 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:04.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:04.639 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:04 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.639 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:04 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.640 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:04 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.640 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:04 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:04 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:04 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:04.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.776 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:04.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:04.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:04.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:04.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:05.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:05.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:05.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:05.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:05.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 24 16 52117 52101 135.652 7.5 0.0025653 0.00709664 2024-03-26T21:44:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:05.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:05.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:05.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: pgmap v708: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: osdmap e344: 8 total, 8 up, 7 in 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: pgmap v708: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:44:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: osdmap e344: 8 total, 8 up, 7 in 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:44:05.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:05 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:44:05.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:05.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: pgmap v708: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: osdmap e344: 8 total, 8 up, 7 in 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:44:05.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:44:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:44:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:05 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:44:05.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:05.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:05.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:06.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:44:06.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:06.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:06.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:06.258 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 25 15 53609 53594 133.959 93.3125 0.00720235 0.00745997 2024-03-26T21:44:06.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:06.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:06.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.519 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:06.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:44:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:44:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: osdmap e345: 8 total, 8 up, 7 in 2024-03-26T21:44:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 10.5 scrub starts 2024-03-26T21:44:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 10.2 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 10.9 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[29406]: 10.1 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: osdmap e345: 8 total, 8 up, 7 in 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 10.5 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 10.2 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 10.9 scrub starts 2024-03-26T21:44:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:06 smithi055 ceph-mon[40587]: 10.1 scrub starts 2024-03-26T21:44:06.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:06.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:06.733 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: osdmap e345: 8 total, 8 up, 7 in 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 10.5 scrub starts 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 10.2 scrub starts 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 10.9 scrub starts 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T21:44:06.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:06 smithi067 ceph-mon[33346]: 10.1 scrub starts 2024-03-26T21:44:06.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:06.755 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:44:06.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:06.776 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:44:06.799 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:44:06.813 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:06.836 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:44:06.860 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:06.867 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:44:06.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:06.892 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:44:06.898 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:06.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:06.912 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:44:06.914 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:44:07.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:07.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.148 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:07.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:07.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 26 16 53802 53786 129.268 12 0.00170702 0.00746874 2024-03-26T21:44:07.273 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:07.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:07.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: pgmap v711: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:44:07.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.5 scrub ok 2024-03-26T21:44:07.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.0 scrub starts 2024-03-26T21:44:07.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.2 scrub ok 2024-03-26T21:44:07.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.9 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.1 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.0 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.4 scrub starts 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: osdmap e346: 8 total, 8 up, 7 in 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.3 deep-scrub starts 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.6 scrub starts 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.4 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 10.6 scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/437204941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:44:07.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[40587]: osdmap e347: 8 total, 8 up, 7 in 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: pgmap v711: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.5 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.0 scrub starts 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.2 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.9 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.1 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.0 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.4 scrub starts 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: osdmap e346: 8 total, 8 up, 7 in 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.3 deep-scrub starts 2024-03-26T21:44:07.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.6 scrub starts 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.4 scrub ok 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 10.6 scrub ok 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/437204941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:44:07.387 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:07 smithi055 ceph-mon[29406]: osdmap e347: 8 total, 8 up, 7 in 2024-03-26T21:44:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:07.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:07.525 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:07.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:07.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: pgmap v711: 158 pgs: 10 unknown, 148 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T21:44:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.5 scrub ok 2024-03-26T21:44:07.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.0 scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.2 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.9 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.1 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.0 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.4 scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: osdmap e346: 8 total, 8 up, 7 in 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.3 deep-scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.6 scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.4 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 10.6 scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:44:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/437204941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:44:07.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:07 smithi067 ceph-mon[33346]: osdmap e347: 8 total, 8 up, 7 in 2024-03-26T21:44:07.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:07.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:07.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:08.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:08.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:08.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:08.246 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:08.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 27 16 53971 53955 124.872 10.5625 0.00520308 0.00800179 2024-03-26T21:44:08.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:08.391 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:08.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:08.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 10.7 scrub starts 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 10.7 scrub ok 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 10.8 deep-scrub starts 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 10.3 deep-scrub ok 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:44:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[29406]: osdmap e348: 8 total, 8 up, 7 in 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 10.7 scrub starts 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 10.7 scrub ok 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 10.8 deep-scrub starts 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 10.3 deep-scrub ok 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:44:08.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-mon[40587]: osdmap e348: 8 total, 8 up, 7 in 2024-03-26T21:44:08.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:08.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 10.7 scrub starts 2024-03-26T21:44:08.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 10.7 scrub ok 2024-03-26T21:44:08.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 10.8 deep-scrub starts 2024-03-26T21:44:08.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:44:08.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 10.3 deep-scrub ok 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:44:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:08 smithi067 ceph-mon[33346]: osdmap e348: 8 total, 8 up, 7 in 2024-03-26T21:44:08.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:08.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:08.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:08.946 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:08.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:09.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:09.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:09.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:09.224 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:09.257 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 28 16 54996 54980 122.701 64.0625 0.0267715 0.00814321 2024-03-26T21:44:09.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:09.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:09.362 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:09.462 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:44:09.462 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:44:09.462 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-26T21:44:09.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd in 3 2024-03-26T21:44:09.463 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:09.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:09.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:09.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:09 smithi067 ceph-mon[33346]: pgmap v714: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 6 unknown, 146 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-26T21:44:09.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:09 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:44:09.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:09 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:44:09.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:09 smithi067 ceph-mon[33346]: osdmap e349: 8 total, 8 up, 7 in 2024-03-26T21:44:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:09.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:09.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[40587]: pgmap v714: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 6 unknown, 146 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-26T21:44:09.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:44:09.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:44:09.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[40587]: osdmap e349: 8 total, 8 up, 7 in 2024-03-26T21:44:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[29406]: pgmap v714: 158 pgs: 4 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 6 unknown, 146 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-03-26T21:44:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:44:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:44:09.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:09 smithi055 ceph-mon[29406]: osdmap e349: 8 total, 8 up, 7 in 2024-03-26T21:44:09.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:09.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:09.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:10.038 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:10.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:10.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:44:10.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:10.258 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 29 16 56001 55985 120.634 62.8125 0.0111213 0.00828264 2024-03-26T21:44:10.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:10.463 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.520 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:10.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:10.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:10.654 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:10.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:10.700 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:10 smithi055 ceph-mon[29406]: 10.8 deep-scrub ok 2024-03-26T21:44:10.703 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:10 smithi055 ceph-mon[40587]: 10.8 deep-scrub ok 2024-03-26T21:44:10.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:10 smithi067 ceph-mon[33346]: 10.8 deep-scrub ok 2024-03-26T21:44:10.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:10.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:10.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:10.980 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:10.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:11.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:11.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:11.258 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 30 16 56945 56929 118.58 59 0.0223074 0.00842744 2024-03-26T21:44:11.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:11.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:11.412 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:11.417 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:11.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:11.571 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:11 smithi055 ceph-mon[29406]: pgmap v717: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.8 MiB/s, 45 objects/s recovering 2024-03-26T21:44:11.572 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:11 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 1605/162627 objects degraded (0.987%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:44:11.572 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:11 smithi055 ceph-mon[40587]: pgmap v717: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.8 MiB/s, 45 objects/s recovering 2024-03-26T21:44:11.572 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:11 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 1605/162627 objects degraded (0.987%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:44:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:11.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:11.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:11.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:11 smithi067 ceph-mon[33346]: pgmap v717: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.8 MiB/s, 45 objects/s recovering 2024-03-26T21:44:11.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:11 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 1605/162627 objects degraded (0.987%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:44:11.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:11.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:11.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:11.982 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:11.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:12.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:12.128 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:12.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:12.258 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 31 16 58051 58035 116.984 69.125 0.0162646 0.00854261 2024-03-26T21:44:12.262 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:12.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:12.388 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:12.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:12.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:12.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:12.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:12.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:12.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:12.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:13.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:13.056 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[40587]: pgmap v718: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 970 B/s rd, 0 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T21:44:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[40587]: osdmap e350: 8 total, 8 up, 7 in 2024-03-26T21:44:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:44:13.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:44:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[29406]: pgmap v718: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 970 B/s rd, 0 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T21:44:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[29406]: osdmap e350: 8 total, 8 up, 7 in 2024-03-26T21:44:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:44:13.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:12 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:44:13.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:13.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:12 smithi067 ceph-mon[33346]: pgmap v718: 158 pgs: 4 active+recovering+undersized+remapped, 5 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 145 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 970 B/s rd, 0 op/s; 1605/162627 objects degraded (0.987%); 13242/162627 objects misplaced (8.143%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T21:44:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:12 smithi067 ceph-mon[33346]: osdmap e350: 8 total, 8 up, 7 in 2024-03-26T21:44:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:12 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:44:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:12 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:44:13.258 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 32 16 59055 59039 115.289 62.75 0.0216084 0.00866685 2024-03-26T21:44:13.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:13.339 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:13.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:13.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:13.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:13.626 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:13.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:13.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:13.843 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:44:13.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:13.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:13.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:14.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:14.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:14.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:13 smithi067 ceph-mon[33346]: osdmap e351: 8 total, 8 up, 7 in 2024-03-26T21:44:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:13 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3908013131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:13 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 33 16 60010 59994 113.596 59.6875 0.00916319 0.00879605 2024-03-26T21:44:14.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[40587]: osdmap e351: 8 total, 8 up, 7 in 2024-03-26T21:44:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3908013131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[29406]: osdmap e351: 8 total, 8 up, 7 in 2024-03-26T21:44:14.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3908013131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:13 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:44:14.324 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:14.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:14.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:14.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:14.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:14.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:14.755 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:14.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:14.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:14.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:14.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:15.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:15.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:14 smithi067 ceph-mon[33346]: pgmap v721: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 6.1 MiB/s wr, 100 op/s; 6626/170877 objects degraded (3.878%); 24198/170877 objects misplaced (14.161%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:44:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:14 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:44:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:14 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:44:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:14 smithi067 ceph-mon[33346]: osdmap e352: 8 total, 8 up, 7 in 2024-03-26T21:44:15.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 34 16 61037 61021 112.143 64.1875 0.0755512 0.00890401 2024-03-26T21:44:15.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:15.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[40587]: pgmap v721: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 6.1 MiB/s wr, 100 op/s; 6626/170877 objects degraded (3.878%); 24198/170877 objects misplaced (14.161%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:44:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:44:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:44:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[40587]: osdmap e352: 8 total, 8 up, 7 in 2024-03-26T21:44:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[29406]: pgmap v721: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 147 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 6.1 MiB/s wr, 100 op/s; 6626/170877 objects degraded (3.878%); 24198/170877 objects misplaced (14.161%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:44:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:44:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:44:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:14 smithi055 ceph-mon[29406]: osdmap e352: 8 total, 8 up, 7 in 2024-03-26T21:44:15.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:15.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:15.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:15.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:15.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:15.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:15.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:15.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:15.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:15.998 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:15.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:16.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:16.142 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:16.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:15 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/175662 objects degraded (3.772%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:16.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:16.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 35 16 61924 61908 110.523 55.4375 0.00995629 0.00904248 2024-03-26T21:44:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:16.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:15 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/175662 objects degraded (3.772%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:16.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:15 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/175662 objects degraded (3.772%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:16.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:16.415 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:16.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:16.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:16.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:16.706 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:16.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:16.944 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:16.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:16.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:17.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:17.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:16 smithi067 ceph-mon[33346]: pgmap v723: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 6626/175662 objects degraded (3.772%); 22338/175662 objects misplaced (12.716%); 20 MiB/s, 325 objects/s recovering 2024-03-26T21:44:17.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 36 16 62865 62849 109.086 58.8125 0.00660981 0.00911909 2024-03-26T21:44:17.261 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:16 smithi055 ceph-mon[40587]: pgmap v723: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 6626/175662 objects degraded (3.772%); 22338/175662 objects misplaced (12.716%); 20 MiB/s, 325 objects/s recovering 2024-03-26T21:44:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:16 smithi055 ceph-mon[29406]: pgmap v723: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 6626/175662 objects degraded (3.772%); 22338/175662 objects misplaced (12.716%); 20 MiB/s, 325 objects/s recovering 2024-03-26T21:44:17.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:17.393 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:17.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:17.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:17.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:17.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:17.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.797 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:17.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:17.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:17.938 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:17.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:18.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:18.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:18.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-mon[33346]: osdmap e353: 8 total, 8 up, 7 in 2024-03-26T21:44:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:17 smithi067 ceph-mon[33346]: osdmap e354: 8 total, 8 up, 7 in 2024-03-26T21:44:18.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 37 16 63342 63326 106.944 29.8125 0.0144527 0.00934291 2024-03-26T21:44:18.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[40587]: osdmap e353: 8 total, 8 up, 7 in 2024-03-26T21:44:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:18.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[40587]: osdmap e354: 8 total, 8 up, 7 in 2024-03-26T21:44:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[29406]: osdmap e353: 8 total, 8 up, 7 in 2024-03-26T21:44:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:18.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:18.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:17 smithi055 ceph-mon[29406]: osdmap e354: 8 total, 8 up, 7 in 2024-03-26T21:44:18.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:18.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:18.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:18.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:18.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:18.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:18.757 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:18.860 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:18.904 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:18.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:19.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.051 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:19.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:19.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:19.202 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:19.244 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:19.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 38 16 64277 64261 105.668 58.4375 0.00645474 0.00945014 2024-03-26T21:44:19.304 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:19.328 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:19.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:19.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:19.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:19 smithi055 ceph-mon[40587]: pgmap v725: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 880 B/s rd, 27 MiB/s wr, 439 op/s; 6626/183951 objects degraded (3.602%); 21539/183951 objects misplaced (11.709%); 25 MiB/s, 404 objects/s recovering 2024-03-26T21:44:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:19 smithi055 ceph-mon[29406]: pgmap v725: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 880 B/s rd, 27 MiB/s wr, 439 op/s; 6626/183951 objects degraded (3.602%); 21539/183951 objects misplaced (11.709%); 25 MiB/s, 404 objects/s recovering 2024-03-26T21:44:19.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:19.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:19.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:19.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:19 smithi067 ceph-mon[33346]: pgmap v725: 158 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 880 B/s rd, 27 MiB/s wr, 439 op/s; 6626/183951 objects degraded (3.602%); 21539/183951 objects misplaced (11.709%); 25 MiB/s, 404 objects/s recovering 2024-03-26T21:44:19.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:19.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:19.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:19.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:20.010 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:20.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:20.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:20.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:20.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 39 16 65169 65153 104.388 55.75 0.0147474 0.00957231 2024-03-26T21:44:20.274 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:20.376 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:20.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:20.507 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:20.548 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:20.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:20.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:20.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:20.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:20.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:20.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:21.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:21.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:21.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:21.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:44:21.269852+0000 min lat: 0.00140536 max lat: 1.79079 avg lat: 0.0096986 2024-03-26T21:44:21.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:44:21.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 40 16 65969 65953 103.028 50 0.00407242 0.0096986 2024-03-26T21:44:21.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:21.334 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:21.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:21.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:21.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:21 smithi055 ceph-mon[29406]: pgmap v727: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 758 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 38 MiB/s, 603 objects/s recovering 2024-03-26T21:44:21.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:21 smithi055 ceph-mon[40587]: pgmap v727: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 758 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 38 MiB/s, 603 objects/s recovering 2024-03-26T21:44:21.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:21.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:21.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:21.733 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:21.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:21 smithi067 ceph-mon[33346]: pgmap v727: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 758 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 38 MiB/s, 603 objects/s recovering 2024-03-26T21:44:21.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:21.881 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:21.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:21.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:22.005 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:22.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:22.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:22.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:22.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 41 15 66945 66930 102.004 61.0625 0.0593733 0.00979816 2024-03-26T21:44:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.293 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:22.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:22.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:22.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:22.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:22.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/189696 objects degraded (3.493%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:22.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/189696 objects degraded (3.493%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:22.576 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:22.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:22.721 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/189696 objects degraded (3.493%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:22.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:22.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:22.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:23.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.114 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:23.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:23.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:23.261 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 42 16 67815 67799 100.869 54.3125 0.00386423 0.00990808 2024-03-26T21:44:23.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:23.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:23.490 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.537 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:23.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:23 smithi055 ceph-mon[40587]: pgmap v728: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 657 B/s rd, 38 MiB/s wr, 601 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:23.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:23 smithi055 ceph-mon[29406]: pgmap v728: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 657 B/s rd, 38 MiB/s wr, 601 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:23.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:23.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:23.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:23 smithi067 ceph-mon[33346]: pgmap v728: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 657 B/s rd, 38 MiB/s wr, 601 op/s; 6626/189696 objects degraded (3.493%); 19179/189696 objects misplaced (10.110%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:23.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:23.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:23.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:23.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:23.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:24.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.066 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:24.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:24.212 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:24.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 43 16 68682 68666 99.7834 54.1875 0.00505331 0.0100146 2024-03-26T21:44:24.314 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.357 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:24.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:24.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:24.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:24.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:24.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:24.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:24.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:24.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:25.069 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:25.092 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:25.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:25.240 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:25.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 44 16 69586 69570 98.7998 56.5 0.00535124 0.0101158 2024-03-26T21:44:25.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:25.373 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:25.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.520 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:25.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:25.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:25 smithi055 ceph-mon[40587]: pgmap v729: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 940 op/s; 6626/198204 objects degraded (3.343%); 17990/198204 objects misplaced (9.077%); 23 MiB/s, 374 objects/s recovering 2024-03-26T21:44:25.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:25 smithi055 ceph-mon[29406]: pgmap v729: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 940 op/s; 6626/198204 objects degraded (3.343%); 17990/198204 objects misplaced (9.077%); 23 MiB/s, 374 objects/s recovering 2024-03-26T21:44:25.622 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:25.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:25.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:25 smithi067 ceph-mon[33346]: pgmap v729: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 940 op/s; 6626/198204 objects degraded (3.343%); 17990/198204 objects misplaced (9.077%); 23 MiB/s, 374 objects/s recovering 2024-03-26T21:44:25.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:25.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:25.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:25.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:25.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:26.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:26.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:26.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:26.200 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:26.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 45 16 70536 70520 97.9237 59.375 0.00968383 0.0102069 2024-03-26T21:44:26.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:26.336 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:26.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:26.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:26.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:26.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:26.741 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:26.843 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:26.876 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:26.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:26.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:27.001 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:27.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:27.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:27.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:27.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 46 16 71475 71459 97.0704 58.6875 0.00787623 0.0102954 2024-03-26T21:44:27.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:27.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:27.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:27 smithi055 ceph-mon[29406]: pgmap v730: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 839 op/s; 6626/202251 objects degraded (3.276%); 16893/202251 objects misplaced (8.352%); 26 MiB/s, 416 objects/s recovering 2024-03-26T21:44:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/202251 objects degraded (3.276%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:27 smithi055 ceph-mon[40587]: pgmap v730: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 839 op/s; 6626/202251 objects degraded (3.276%); 16893/202251 objects misplaced (8.352%); 26 MiB/s, 416 objects/s recovering 2024-03-26T21:44:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/202251 objects degraded (3.276%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:27.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:27.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:27 smithi067 ceph-mon[33346]: pgmap v730: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 839 op/s; 6626/202251 objects degraded (3.276%); 16893/202251 objects misplaced (8.352%); 26 MiB/s, 416 objects/s recovering 2024-03-26T21:44:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/202251 objects degraded (3.276%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:27.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:27.791 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:27.834 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:27.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:27.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:27.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:28.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.106 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:28.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:28.207 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:28.250 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:28.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 47 16 72333 72317 96.146 53.625 0.0182741 0.0103957 2024-03-26T21:44:28.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:28.398 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:44:28.398 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:44:28.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:28.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:28.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:28.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:28.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:28.657 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:28.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:28.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.793 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:28.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:28.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:28.920 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:28.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:29.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.057 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:29.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:29.191 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:29.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 48 16 73357 73341 95.4763 64 0.00528552 0.0104682 2024-03-26T21:44:29.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:29.300 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:29.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:29.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:29.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:29.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:29 smithi055 ceph-mon[40587]: pgmap v731: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 868 op/s; 6626/209982 objects degraded (3.156%); 16081/209982 objects misplaced (7.658%); 23 MiB/s, 364 objects/s recovering 2024-03-26T21:44:29.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:29 smithi055 ceph-mon[29406]: pgmap v731: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 868 op/s; 6626/209982 objects degraded (3.156%); 16081/209982 objects misplaced (7.658%); 23 MiB/s, 364 objects/s recovering 2024-03-26T21:44:29.591 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:29.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:29.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:29 smithi067 ceph-mon[33346]: pgmap v731: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 868 op/s; 6626/209982 objects degraded (3.156%); 16081/209982 objects misplaced (7.658%); 23 MiB/s, 364 objects/s recovering 2024-03-26T21:44:29.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:29.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:29.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:29.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:30.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:30.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:30.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 49 16 74196 74180 94.598 52.4375 0.0240624 0.0105659 2024-03-26T21:44:30.264 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:30.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:30.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:30.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:30.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:30.670 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:44:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:44:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:30 smithi067 ceph-mon[33346]: osdmap e355: 8 total, 8 up, 7 in 2024-03-26T21:44:30.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:30 smithi067 ceph-mon[33346]: pgmap v733: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:30.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:44:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:44:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[29406]: osdmap e355: 8 total, 8 up, 7 in 2024-03-26T21:44:30.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[29406]: pgmap v733: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:44:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:44:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[40587]: osdmap e355: 8 total, 8 up, 7 in 2024-03-26T21:44:30.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:30 smithi055 ceph-mon[40587]: pgmap v733: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:30.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:30.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:30.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:31.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:31.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:31.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:31.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:31.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 50 16 75095 75079 93.8297 56.1875 0.00899923 0.0106382 2024-03-26T21:44:31.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:31.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:31.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:31.596 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:31.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:31.705 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:31 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:31.706 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:31 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:31.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:31 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:44:31.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:31.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:31.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:31.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:32.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:32.111 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:32.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:32.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:32.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 51 16 75665 75649 92.6886 35.625 0.00323217 0.0107277 2024-03-26T21:44:32.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:32.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:32.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:32.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:32.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:32.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:32.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:32 smithi067 ceph-mon[33346]: pgmap v734: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/216042 objects degraded (3.067%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:32 smithi067 ceph-mon[33346]: osdmap e356: 8 total, 8 up, 7 in 2024-03-26T21:44:32.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[29406]: pgmap v734: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/216042 objects degraded (3.067%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[29406]: osdmap e356: 8 total, 8 up, 7 in 2024-03-26T21:44:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[40587]: pgmap v734: 158 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 878 op/s; 6626/216042 objects degraded (3.067%); 14560/216042 objects misplaced (6.739%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:44:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/216042 objects degraded (3.067%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[40587]: osdmap e356: 8 total, 8 up, 7 in 2024-03-26T21:44:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:32.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:32.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:32.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:33.006 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:33.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:33.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:33.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:33.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 52 16 75748 75732 91.006 5.1875 0.00256766 0.0107223 2024-03-26T21:44:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:33.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:33.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:33.531 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:33.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:33.664 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:33.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:33.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:33.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:34.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:34.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:44:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:34.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:33 smithi055 ceph-mon[29406]: osdmap e357: 8 total, 8 up, 7 in 2024-03-26T21:44:34.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:33 smithi055 ceph-mon[40587]: osdmap e357: 8 total, 8 up, 7 in 2024-03-26T21:44:34.161 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:34.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:33 smithi067 ceph-mon[33346]: osdmap e357: 8 total, 8 up, 7 in 2024-03-26T21:44:34.262 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 53 15 76589 76574 90.2818 52.625 0.0281973 0.0110715 2024-03-26T21:44:34.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:34.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:34.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:34.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:34.619 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:34.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:34.763 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:34.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:34.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:34.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:34.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:35.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:35.139 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:35.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:34 smithi067 ceph-mon[33346]: pgmap v737: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 52 MiB/s wr, 833 op/s; 6626/224961 objects degraded (2.945%); 13027/224961 objects misplaced (5.791%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:44:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:34 smithi067 ceph-mon[33346]: osdmap e358: 8 total, 8 up, 7 in 2024-03-26T21:44:35.263 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 54 15 77766 77751 89.9722 73.5625 0.0106048 0.0111093 2024-03-26T21:44:35.264 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:35.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:35.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:34 smithi055 ceph-mon[40587]: pgmap v737: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 52 MiB/s wr, 833 op/s; 6626/224961 objects degraded (2.945%); 13027/224961 objects misplaced (5.791%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:44:35.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:34 smithi055 ceph-mon[40587]: osdmap e358: 8 total, 8 up, 7 in 2024-03-26T21:44:35.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:34 smithi055 ceph-mon[29406]: pgmap v737: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 52 MiB/s wr, 833 op/s; 6626/224961 objects degraded (2.945%); 13027/224961 objects misplaced (5.791%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:44:35.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:34 smithi055 ceph-mon[29406]: osdmap e358: 8 total, 8 up, 7 in 2024-03-26T21:44:35.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:35.444 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:35.546 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.579 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:35.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:35.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.724 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:35.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:35.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:35.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:35.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:36.004 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:36.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.145 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:36.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:36.263 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 55 15 78946 78931 89.6771 73.75 0.00775907 0.0111459 2024-03-26T21:44:36.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:36.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:36.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:36.553 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:36.658 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:36.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.835 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:36.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:36.915 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:36 smithi055 ceph-mon[40587]: pgmap v739: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 648 op/s; 6626/227706 objects degraded (2.910%); 11613/227706 objects misplaced (5.100%); 17 MiB/s, 279 objects/s recovering 2024-03-26T21:44:36.916 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:36 smithi055 ceph-mon[29406]: pgmap v739: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 648 op/s; 6626/227706 objects degraded (2.910%); 11613/227706 objects misplaced (5.100%); 17 MiB/s, 279 objects/s recovering 2024-03-26T21:44:36.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:36.976 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:36.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:37.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:37.112 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:37.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:36 smithi067 ceph-mon[33346]: pgmap v739: 158 pgs: 1 creating+peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 648 op/s; 6626/227706 objects degraded (2.910%); 11613/227706 objects misplaced (5.100%); 17 MiB/s, 279 objects/s recovering 2024-03-26T21:44:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:36 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/227706 objects degraded (2.910%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:37.263 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 56 16 79840 79824 89.0711 55.8125 0.00829751 0.0112212 2024-03-26T21:44:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:37.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:37.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:36 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/227706 objects degraded (2.910%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:37.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:36 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/227706 objects degraded (2.910%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:37.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.405 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:37.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:37.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:37.537 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:37.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:37.702 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:37.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:37.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:37.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:37.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:38.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:38.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:38.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:37 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:37 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:37 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:37 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:38.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 57 16 80919 80903 88.6915 67.4375 0.00372764 0.0112641 2024-03-26T21:44:38.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:44:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:44:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:37 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:38.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:38.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:38.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:38.601 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:38.650 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:38.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.794 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:38.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:38.896 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:38.920 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:38.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:39.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:39.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:39.140 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:38 smithi067 ceph-mon[33346]: pgmap v740: 158 pgs: 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 6626/235149 objects degraded (2.818%); 9897/235149 objects misplaced (4.209%); 26 MiB/s, 422 objects/s recovering 2024-03-26T21:44:39.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:39.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:39.236 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:38 smithi055 ceph-mon[29406]: pgmap v740: 158 pgs: 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 6626/235149 objects degraded (2.818%); 9897/235149 objects misplaced (4.209%); 26 MiB/s, 422 objects/s recovering 2024-03-26T21:44:39.236 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:39.236 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:38 smithi055 ceph-mon[40587]: pgmap v740: 158 pgs: 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 6626/235149 objects degraded (2.818%); 9897/235149 objects misplaced (4.209%); 26 MiB/s, 422 objects/s recovering 2024-03-26T21:44:39.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 58 16 81964 81948 88.2884 65.3125 0.0102937 0.0113212 2024-03-26T21:44:39.305 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:39.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:39.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:39.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.630 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:39.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:39.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:39.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:39.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:39.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:39.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:40.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:40.058 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:40.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:40.196 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:40.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 59 16 83131 83115 88.0281 72.9375 0.00993193 0.0113547 2024-03-26T21:44:40.298 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:40.339 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:40.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:40.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:40.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:40.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:40.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:40.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:40.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:40.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:41.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:41.160 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:40 smithi067 ceph-mon[33346]: pgmap v741: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 907 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:44:41.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:41.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:44:41.272515+0000 min lat: 0.00140536 max lat: 1.79079 avg lat: 0.0113723 2024-03-26T21:44:41.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:44:41.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 60 16 84153 84137 87.6255 63.875 0.00786969 0.0113723 2024-03-26T21:44:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:41.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:41.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:40 smithi055 ceph-mon[40587]: pgmap v741: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 907 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:44:41.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:40 smithi055 ceph-mon[29406]: pgmap v741: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 907 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:44:41.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:41.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:41.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:41.563 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:41.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:41.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:41.820 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:41.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:41.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:41.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:42.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:42.083 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:42.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:42.213 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:41 smithi067 ceph-mon[33346]: osdmap e359: 8 total, 8 up, 7 in 2024-03-26T21:44:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:41 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6626/241290 objects degraded (2.746%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:41 smithi067 ceph-mon[33346]: osdmap e360: 8 total, 8 up, 7 in 2024-03-26T21:44:42.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 61 16 84571 84555 86.6173 26.125 0.00537165 0.011482 2024-03-26T21:44:42.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[40587]: osdmap e359: 8 total, 8 up, 7 in 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6626/241290 objects degraded (2.746%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[40587]: osdmap e360: 8 total, 8 up, 7 in 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[29406]: osdmap e359: 8 total, 8 up, 7 in 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6626/241290 objects degraded (2.746%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:42.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:41 smithi055 ceph-mon[29406]: osdmap e360: 8 total, 8 up, 7 in 2024-03-26T21:44:42.340 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:42.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:42.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:42.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:42.599 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:42.700 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:42.732 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:42.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:42.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:42.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:43.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:43.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:43.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:42 smithi067 ceph-mon[33346]: pgmap v743: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 680 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 15 MiB/s, 246 objects/s recovering 2024-03-26T21:44:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:42 smithi067 ceph-mon[33346]: osdmap e361: 8 total, 8 up, 7 in 2024-03-26T21:44:43.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:43.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 62 16 85030 85014 85.683 28.6875 0.00849971 0.0116657 2024-03-26T21:44:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:43.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:43.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:42 smithi055 ceph-mon[29406]: pgmap v743: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 680 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 15 MiB/s, 246 objects/s recovering 2024-03-26T21:44:43.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:42 smithi055 ceph-mon[29406]: osdmap e361: 8 total, 8 up, 7 in 2024-03-26T21:44:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:42 smithi055 ceph-mon[40587]: pgmap v743: 158 pgs: 4 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 680 op/s; 6626/241290 objects degraded (2.746%); 8650/241290 objects misplaced (3.585%); 15 MiB/s, 246 objects/s recovering 2024-03-26T21:44:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:42 smithi055 ceph-mon[40587]: osdmap e361: 8 total, 8 up, 7 in 2024-03-26T21:44:43.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:43.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:43.546 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:43.588 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:43.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:43.716 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:43.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:43.841 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:43.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:43.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:44.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:44.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:44.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:44.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 63 16 86159 86143 85.4429 70.5625 0.0267931 0.011698 2024-03-26T21:44:44.358 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:44.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:44.505 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:44.544 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:44.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:44.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:44.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:44.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:44.958 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:44.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:45.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:45.084 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:45.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:45.221 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:45.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:44 smithi067 ceph-mon[33346]: pgmap v746: 158 pgs: 1 creating+peering, 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 932 op/s; 1605/251916 objects degraded (0.637%); 11637/251916 objects misplaced (4.619%); 16 MiB/s, 253 objects/s recovering 2024-03-26T21:44:45.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 64 15 87361 87346 85.2826 75.1875 0.0114609 0.0117208 2024-03-26T21:44:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:44 smithi055 ceph-mon[40587]: pgmap v746: 158 pgs: 1 creating+peering, 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 932 op/s; 1605/251916 objects degraded (0.637%); 11637/251916 objects misplaced (4.619%); 16 MiB/s, 253 objects/s recovering 2024-03-26T21:44:45.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:44 smithi055 ceph-mon[29406]: pgmap v746: 158 pgs: 1 creating+peering, 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 932 op/s; 1605/251916 objects degraded (0.637%); 11637/251916 objects misplaced (4.619%); 16 MiB/s, 253 objects/s recovering 2024-03-26T21:44:45.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:45.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:45.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:45.505 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:45.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.638 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:45.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:45.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:45.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:45.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:45.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:45.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:46.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:46.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:46.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:46.188 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:46.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:45 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1605/251916 objects degraded (0.637%), 1 pg degraded) 2024-03-26T21:44:46.229 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:45 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1605/251916 objects degraded (0.637%), 1 pg degraded) 2024-03-26T21:44:46.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:45 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1605/251916 objects degraded (0.637%), 1 pg degraded) 2024-03-26T21:44:46.264 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 65 16 88554 88538 85.1166 74.5 0.0104361 0.0117436 2024-03-26T21:44:46.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:46.325 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:46.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:46.468 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:46.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:46.604 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:46.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:46.740 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:46.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:46.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:46.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:44:46.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:47.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:47.019 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:46 smithi055 ceph-mon[40587]: pgmap v747: 158 pgs: 1 creating+peering, 1 peering, 4 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 11731/254958 objects misplaced (4.601%); 16 MiB/s, 259 objects/s recovering 2024-03-26T21:44:47.019 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:46 smithi055 ceph-mon[40587]: osdmap e362: 8 total, 8 up, 7 in 2024-03-26T21:44:47.019 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:46 smithi055 ceph-mon[29406]: pgmap v747: 158 pgs: 1 creating+peering, 1 peering, 4 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 11731/254958 objects misplaced (4.601%); 16 MiB/s, 259 objects/s recovering 2024-03-26T21:44:47.019 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:46 smithi055 ceph-mon[29406]: osdmap e362: 8 total, 8 up, 7 in 2024-03-26T21:44:47.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:47.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:47.183 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:44:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:46 smithi067 ceph-mon[33346]: pgmap v747: 158 pgs: 1 creating+peering, 1 peering, 4 active+recovering+undersized+remapped, 152 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 11731/254958 objects misplaced (4.601%); 16 MiB/s, 259 objects/s recovering 2024-03-26T21:44:47.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:46 smithi067 ceph-mon[33346]: osdmap e362: 8 total, 8 up, 7 in 2024-03-26T21:44:47.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:47.265 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 66 16 89636 89620 84.8504 67.625 0.0279071 0.0117797 2024-03-26T21:44:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.315 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:47.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:47.416 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:47.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:47.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:47.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:47.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:47.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:47.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:47.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:47.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:47.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:47 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:44:47.683+0000 7fb1ff446700 -1 mon.a@0(leader).osd e362 definitely_dead 0 2024-03-26T21:44:47.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:48.021 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:48.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:48.165 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:48.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 67 16 90644 90628 84.5242 63 0.00381288 0.011825 2024-03-26T21:44:48.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:48.301 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:48.325 INFO:teuthology.orchestra.run.smithi055.stderr:marked in osd.3. 2024-03-26T21:44:48.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:48.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:48.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:48.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1227134813' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1227134813' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:48.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:44:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:44:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:48 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1227134813' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:48 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T21:44:48.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:48.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:48.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:48.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:49.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:49.064 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:49.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:49.190 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:49.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 68 16 90714 90698 83.3457 4.375 0.00724798 0.0118335 2024-03-26T21:44:49.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:49.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.462 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:49.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:49.563 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:49.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:49.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:49 smithi067 ceph-mon[33346]: pgmap v749: 158 pgs: 1 peering, 4 active+recovering+undersized+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 10656/263733 objects misplaced (4.040%); 21 MiB/s, 333 objects/s recovering 2024-03-26T21:44:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:49 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T21:44:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:49 smithi067 ceph-mon[33346]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:44:49.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[29406]: pgmap v749: 158 pgs: 1 peering, 4 active+recovering+undersized+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 10656/263733 objects misplaced (4.040%); 21 MiB/s, 333 objects/s recovering 2024-03-26T21:44:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T21:44:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[29406]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:44:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[40587]: pgmap v749: 158 pgs: 1 peering, 4 active+recovering+undersized+remapped, 153 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.25k op/s; 10656/263733 objects misplaced (4.040%); 21 MiB/s, 333 objects/s recovering 2024-03-26T21:44:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T21:44:49.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:49 smithi055 ceph-mon[40587]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T21:44:49.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:49.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:49.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:50.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:50.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:50.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:50.216 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:50.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 69 16 90714 90698 82.1379 0 - 0.0118335 2024-03-26T21:44:50.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:50.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:50.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:50.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:50.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:50.640 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[29406]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[29406]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[40587]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:44:50.702 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:50 smithi055 ceph-mon[40587]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:44:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:50 smithi067 ceph-mon[33346]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T21:44:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:50 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:44:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:50 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:44:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:50 smithi067 ceph-mon[33346]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T21:44:50.742 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:50.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:50.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:50.912 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:50.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:51.013 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:51.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:51.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:51.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:51.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 70 16 90821 90805 81.0602 3.34375 0.0586226 0.0120643 2024-03-26T21:44:51.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:51.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:51.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:51.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:51.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:51.683 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:51 smithi067 ceph-mon[33346]: pgmap v752: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s; 9200/270261 objects misplaced (3.404%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:44:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:51 smithi067 ceph-mon[33346]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 362'5678 2024-03-26T21:44:51.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:51.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:51.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:51 smithi055 ceph-mon[40587]: pgmap v752: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s; 9200/270261 objects misplaced (3.404%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:44:51.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:51 smithi055 ceph-mon[40587]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 362'5678 2024-03-26T21:44:51.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:51 smithi055 ceph-mon[29406]: pgmap v752: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s; 9200/270261 objects misplaced (3.404%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:44:51.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:51 smithi055 ceph-mon[29406]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 362'5678 2024-03-26T21:44:51.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:51.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:51.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:52.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:52.170 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:52.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:52.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 71 16 90821 90805 79.9186 0 - 0.0120643 2024-03-26T21:44:52.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:52.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:52.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:52.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:52.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:52.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:52.722 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:52.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:52.853 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:52.954 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:52.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:52.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:53.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:53.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:53.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:53.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:52 smithi067 ceph-mon[33346]: pgmap v754: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 749 op/s; 9200/270261 objects misplaced (3.404%); 13 MiB/s, 212 objects/s recovering 2024-03-26T21:44:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:52 smithi067 ceph-mon[33346]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:44:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:53.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 72 16 90821 90805 78.8087 0 - 0.0120643 2024-03-26T21:44:53.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:52 smithi055 ceph-mon[40587]: pgmap v754: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 749 op/s; 9200/270261 objects misplaced (3.404%); 13 MiB/s, 212 objects/s recovering 2024-03-26T21:44:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:52 smithi055 ceph-mon[40587]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:44:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:52 smithi055 ceph-mon[29406]: pgmap v754: 158 pgs: 1 active+undersized+remapped, 15 peering, 3 active+recovering+undersized+remapped, 139 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 749 op/s; 9200/270261 objects misplaced (3.404%); 13 MiB/s, 212 objects/s recovering 2024-03-26T21:44:53.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:52 smithi055 ceph-mon[29406]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T21:44:53.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:53.370 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:53.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:53.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:53.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:53.627 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:53.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:53.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:53.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:53.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:54.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:54.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:54.138 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:54.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:53 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 8506/272955 objects degraded (3.116%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:54.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:54.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 73 16 90821 90805 77.7292 0 - 0.0120643 2024-03-26T21:44:54.268 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:54.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:53 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 8506/272955 objects degraded (3.116%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:54.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:53 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 8506/272955 objects degraded (3.116%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T21:44:54.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:54.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:54.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:54.627 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:54.657 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:54.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:54.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:54.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:54.918 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:54.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:55.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:55.045 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:55.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:54 smithi067 ceph-mon[33346]: pgmap v756: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 15 peering, 5 active+recovering+undersized+remapped, 134 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 25 MiB/s wr, 406 op/s; 8506/272955 objects degraded (3.116%); 21245/272955 objects misplaced (7.783%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:44:55.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 74 16 90821 90805 76.679 0 - 0.0120643 2024-03-26T21:44:55.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:55.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:54 smithi055 ceph-mon[40587]: pgmap v756: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 15 peering, 5 active+recovering+undersized+remapped, 134 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 25 MiB/s wr, 406 op/s; 8506/272955 objects degraded (3.116%); 21245/272955 objects misplaced (7.783%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:44:55.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:54 smithi055 ceph-mon[29406]: pgmap v756: 158 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 15 peering, 5 active+recovering+undersized+remapped, 134 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 768 B/s rd, 25 MiB/s wr, 406 op/s; 8506/272955 objects degraded (3.116%); 21245/272955 objects misplaced (7.783%); 13 MiB/s, 215 objects/s recovering 2024-03-26T21:44:55.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:55.426 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:55.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:55.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:55.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:55.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:55.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:55.813 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:55.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:55.946 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:55.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:56.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:56.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:44:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:44:56.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 75 16 91177 91161 75.9533 4.45 0.00295715 0.0130573 2024-03-26T21:44:56.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:56.347 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:56.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:56.475 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:56.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:56.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:56.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:56.739 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:56.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:56.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:56.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:56.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:57.016 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:57.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:57.154 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[40587]: pgmap v757: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 7 peering, 5 active+recovering+undersized+remapped, 141 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 648 B/s rd, 9.0 MiB/s wr, 144 op/s; 11271/272979 objects degraded (4.129%); 19665/272979 objects misplaced (7.204%); 13 MiB/s, 217 objects/s recovering 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[40587]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[40587]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[29406]: pgmap v757: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 7 peering, 5 active+recovering+undersized+remapped, 141 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 648 B/s rd, 9.0 MiB/s wr, 144 op/s; 11271/272979 objects degraded (4.129%); 19665/272979 objects misplaced (7.204%); 13 MiB/s, 217 objects/s recovering 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[29406]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:44:57.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:56 smithi055 ceph-mon[29406]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:44:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:56 smithi067 ceph-mon[33346]: pgmap v757: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 7 peering, 5 active+recovering+undersized+remapped, 141 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 648 B/s rd, 9.0 MiB/s wr, 144 op/s; 11271/272979 objects degraded (4.129%); 19665/272979 objects misplaced (7.204%); 13 MiB/s, 217 objects/s recovering 2024-03-26T21:44:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:56 smithi067 ceph-mon[33346]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T21:44:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:56 smithi067 ceph-mon[33346]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T21:44:57.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:57.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 76 16 91849 91833 75.5064 42 0.00794644 0.0132324 2024-03-26T21:44:57.285 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-26T21:44:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:57.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:57.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:57.433 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:57.535 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:57.572 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:57.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.714 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:57.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:57.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:44:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:44:57.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:57.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:57.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:57.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:58.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:58.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:58.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:57 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:44:58.262 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:58.266 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 77 16 92784 92768 75.2847 58.4375 0.00582016 0.0132724 2024-03-26T21:44:58.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:57 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:44:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:57 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:44:58.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:58.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:58.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.539 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:58.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:58.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:58.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:58.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:44:58.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:44:58.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:44:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:58.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:44:59.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:59.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:44:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.200 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:59.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:44:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:44:58 smithi067 ceph-mon[33346]: pgmap v760: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 682 B/s rd, 14 MiB/s wr, 228 op/s; 15277/274365 objects degraded (5.568%); 15321/274365 objects misplaced (5.584%); 30 MiB/s, 489 objects/s recovering 2024-03-26T21:44:59.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 78 16 93564 93548 74.9445 48.75 0.00286354 0.0133256 2024-03-26T21:44:59.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:44:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:44:58 smithi055 ceph-mon[40587]: pgmap v760: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 682 B/s rd, 14 MiB/s wr, 228 op/s; 15277/274365 objects degraded (5.568%); 15321/274365 objects misplaced (5.584%); 30 MiB/s, 489 objects/s recovering 2024-03-26T21:44:59.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:44:59.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:44:58 smithi055 ceph-mon[29406]: pgmap v760: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 682 B/s rd, 14 MiB/s wr, 228 op/s; 15277/274365 objects degraded (5.568%); 15321/274365 objects misplaced (5.584%); 30 MiB/s, 489 objects/s recovering 2024-03-26T21:44:59.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:44:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:44:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:44:59.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:44:59.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:44:59.468 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:44:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:44:59.570 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:59.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:59.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:44:59.732 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:44:59.834 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:44:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:44:59.881 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:44:59.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:44:59.982 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:00.021 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:00.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:00.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:00.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 79 16 94481 94465 74.7212 57.3125 0.00703102 0.0133776 2024-03-26T21:45:00.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:00.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:00 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 15277/275673 objects degraded (5.542%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:00.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:00 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 15277/275673 objects degraded (5.542%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:00.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:00.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:00 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 15277/275673 objects degraded (5.542%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:00.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:00.582 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:00.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:00.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:00.855 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:00.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:00.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:00.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:01.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:01.119 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:01.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:01.255 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:01.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:45:01.275393+0000 min lat: 0.00140536 max lat: 8.07586 avg lat: 0.0134111 2024-03-26T21:45:01.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:45:01.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 80 16 95436 95420 74.5332 59.6875 0.00998369 0.0134111 2024-03-26T21:45:01.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:01 smithi055 ceph-mon[29406]: pgmap v761: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 MiB/s wr, 169 op/s; 15277/275673 objects degraded (5.542%); 14817/275673 objects misplaced (5.375%); 26 MiB/s, 424 objects/s recovering 2024-03-26T21:45:01.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:01 smithi055 ceph-mon[40587]: pgmap v761: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 MiB/s wr, 169 op/s; 15277/275673 objects degraded (5.542%); 14817/275673 objects misplaced (5.375%); 26 MiB/s, 424 objects/s recovering 2024-03-26T21:45:01.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:01.400 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:01 smithi067 ceph-mon[33346]: pgmap v761: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 MiB/s wr, 169 op/s; 15277/275673 objects degraded (5.542%); 14817/275673 objects misplaced (5.375%); 26 MiB/s, 424 objects/s recovering 2024-03-26T21:45:01.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:01.541 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:01.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:01.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:01.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:01.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:01.963 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:01.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:02.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:02.102 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:02.204 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:02.228 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:02.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 81 16 96280 96264 74.2642 52.75 0.218062 0.0134406 2024-03-26T21:45:02.285 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:45:02.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:45:02.286 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-26T21:45:02.286 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:45:02.286 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:45:02.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-26T21:45:02.310 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:02 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:02.310 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:02 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:02.310 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:02 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:02.310 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:02 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:02.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:02.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:02 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:02 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:02.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:02.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:02.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:02.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:02.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:02.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:02.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:03.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:03.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:03.198 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:03.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 82 16 97162 97146 74.0308 55.125 0.00598511 0.0134992 2024-03-26T21:45:03.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[40587]: pgmap v762: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 12 MiB/s wr, 193 op/s; 15274/277584 objects degraded (5.502%); 14817/277584 objects misplaced (5.338%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:45:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[40587]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:45:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:03.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[29406]: pgmap v762: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 12 MiB/s wr, 193 op/s; 15274/277584 objects degraded (5.502%); 14817/277584 objects misplaced (5.338%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:45:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[29406]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:45:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:03.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:03.339 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:03.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:03 smithi067 ceph-mon[33346]: pgmap v762: 158 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 148 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 12 MiB/s wr, 193 op/s; 15274/277584 objects degraded (5.502%); 14817/277584 objects misplaced (5.338%); 22 MiB/s, 358 objects/s recovering 2024-03-26T21:45:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:03 smithi067 ceph-mon[33346]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T21:45:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:03 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:03.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:03.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:03.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:03.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:03.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:03.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:03.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:04.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:04.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:04.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:04.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 83 16 98008 97992 73.7758 52.875 0.00620675 0.0135452 2024-03-26T21:45:04.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3666391482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3666391482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:04.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:04.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:04.457 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:04 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:04 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3666391482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:04.559 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:04.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:04.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:04.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:04.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:04.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:04.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:05.002 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:45:05.002 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:45:05.003 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:45:05.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:45:05.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:45:05.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:45:05.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:05.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:05.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:05.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 84 16 98960 98944 73.6058 59.5 0.00649367 0.0135679 2024-03-26T21:45:05.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:05.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[40587]: pgmap v764: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 40 MiB/s wr, 647 op/s; 15274/288111 objects degraded (5.301%); 14094/288111 objects misplaced (4.892%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:45:05.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:45:05.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[40587]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:45:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[29406]: pgmap v764: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 40 MiB/s wr, 647 op/s; 15274/288111 objects degraded (5.301%); 14094/288111 objects misplaced (4.892%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:45:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:45:05.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:05 smithi055 ceph-mon[29406]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:45:05.385 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:05.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:05 smithi067 ceph-mon[33346]: pgmap v764: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 40 MiB/s wr, 647 op/s; 15274/288111 objects degraded (5.301%); 14094/288111 objects misplaced (4.892%); 24 MiB/s, 391 objects/s recovering 2024-03-26T21:45:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:05 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:45:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:05 smithi067 ceph-mon[33346]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T21:45:05.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:05.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:05.694 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:05.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.833 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:05.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:05.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:05.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:06.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.106 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:06.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:06.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:06.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 85 16 99750 99734 73.3207 49.375 0.0181765 0.0136306 2024-03-26T21:45:06.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:06.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:06.488 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:06 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 15274/288990 objects degraded (5.285%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:06 smithi067 ceph-mon[33346]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:45:06.526 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:06.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:06 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 15274/288990 objects degraded (5.285%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:06.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:06 smithi055 ceph-mon[40587]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:45:06.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:06 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 15274/288990 objects degraded (5.285%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:06 smithi055 ceph-mon[29406]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T21:45:06.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:06.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:06.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:06.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:06.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:07.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:07.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:07.192 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:07.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 86 16 100439 100423 72.9689 43.0625 0.00468778 0.0136951 2024-03-26T21:45:07.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:07.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:07.399 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:07 smithi067 ceph-mon[33346]: pgmap v766: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 610 op/s; 15274/288990 objects degraded (5.285%); 14094/288990 objects misplaced (4.877%); 12 MiB/s, 189 objects/s recovering 2024-03-26T21:45:07.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:07 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:07.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:07 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:07.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:07 smithi067 ceph-mon[33346]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:45:07.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:07.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[40587]: pgmap v766: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 610 op/s; 15274/288990 objects degraded (5.285%); 14094/288990 objects misplaced (4.877%); 12 MiB/s, 189 objects/s recovering 2024-03-26T21:45:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[40587]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:45:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[29406]: pgmap v766: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 610 op/s; 15274/288990 objects degraded (5.285%); 14094/288990 objects misplaced (4.877%); 12 MiB/s, 189 objects/s recovering 2024-03-26T21:45:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:07 smithi055 ceph-mon[29406]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T21:45:07.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:07.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:07.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:07.740 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:07.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:07.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:07.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:08.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:08.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:08.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 87 16 101378 101362 72.8047 58.6875 0.00567854 0.0137287 2024-03-26T21:45:08.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:08.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:08.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:08.546 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.587 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:08.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:08.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:08.711 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:08.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:08.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:08.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:08.988 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:08.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:09.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:09.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:09.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:09.267 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 88 16 102254 102238 72.5995 54.75 0.00792021 0.0137628 2024-03-26T21:45:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:09 smithi055 ceph-mon[40587]: pgmap v769: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 917 B/s rd, 54 MiB/s wr, 864 op/s; 15274/298533 objects degraded (5.116%); 13647/298533 objects misplaced (4.571%); 15 MiB/s, 247 objects/s recovering 2024-03-26T21:45:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:09 smithi055 ceph-mon[29406]: pgmap v769: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 917 B/s rd, 54 MiB/s wr, 864 op/s; 15274/298533 objects degraded (5.116%); 13647/298533 objects misplaced (4.571%); 15 MiB/s, 247 objects/s recovering 2024-03-26T21:45:09.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:09.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:09.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:09 smithi067 ceph-mon[33346]: pgmap v769: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 917 B/s rd, 54 MiB/s wr, 864 op/s; 15274/298533 objects degraded (5.116%); 13647/298533 objects misplaced (4.571%); 15 MiB/s, 247 objects/s recovering 2024-03-26T21:45:09.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:09.536 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:09.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:09.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:09.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:09.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:09.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:09.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:10.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:10.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:10.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:10.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:10.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 89 16 103167 103151 72.4249 57.0625 0.0152866 0.0138024 2024-03-26T21:45:10.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.382 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:10.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:10.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:10.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:10.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:10.759 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:10.799 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:10.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:10.943 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:10.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:11.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.080 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:11.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:11.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:11.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:11.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 90 9 104046 104037 72.2354 55.375 0.0396858 0.0138298 2024-03-26T21:45:11.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:11.351 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:11.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:11.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:11 smithi067 ceph-mon[33346]: pgmap v770: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 15274/301767 objects degraded (5.062%); 13331/301767 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T21:45:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:11 smithi055 ceph-mon[40587]: pgmap v770: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 15274/301767 objects degraded (5.062%); 13331/301767 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T21:45:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:11 smithi055 ceph-mon[29406]: pgmap v770: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 47 MiB/s wr, 759 op/s; 15274/301767 objects degraded (5.062%); 13331/301767 objects misplaced (4.418%); 11 MiB/s, 173 objects/s recovering 2024-03-26T21:45:11.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:11.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:11.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:11.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:11.863 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:11.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:11.990 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:11.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:12.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:12.217 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:12.244 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total time run: 90.1751 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total writes made: 104046 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Write size: 65536 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object size: 65536 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Bandwidth (MB/sec): 72.1139 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Bandwidth: 49.2231 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max bandwidth (MB/sec): 222.562 2024-03-26T21:45:12.268 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min bandwidth (MB/sec): 0 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average IOPS: 1153 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev IOPS: 787.581 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max IOPS: 3561 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min IOPS: 0 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average Latency(s): 0.0138443 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Latency(s): 0.0884185 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max latency(s): 8.07586 2024-03-26T21:45:12.269 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min latency(s): 0.00140536 2024-03-26T21:45:12.274 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T21:45:12.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:12.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:12.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:12 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 15274/301767 objects degraded (5.062%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:12.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:12.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:12 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 15274/301767 objects degraded (5.062%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:12.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:12 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 15274/301767 objects degraded (5.062%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:12.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:12.654 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:12.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:12.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:12.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:12.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:13.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:13.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:13.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:13.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:13.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:13.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:13.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:13 smithi067 ceph-mon[33346]: pgmap v771: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 676 B/s rd, 40 MiB/s wr, 646 op/s; 15274/303657 objects degraded (5.030%); 13331/303657 objects misplaced (4.390%); 8.6 MiB/s, 137 objects/s recovering 2024-03-26T21:45:13.495 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:13 smithi055 ceph-mon[40587]: pgmap v771: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 676 B/s rd, 40 MiB/s wr, 646 op/s; 15274/303657 objects degraded (5.030%); 13331/303657 objects misplaced (4.390%); 8.6 MiB/s, 137 objects/s recovering 2024-03-26T21:45:13.496 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:13 smithi055 ceph-mon[29406]: pgmap v771: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 676 B/s rd, 40 MiB/s wr, 646 op/s; 15274/303657 objects degraded (5.030%); 13331/303657 objects misplaced (4.390%); 8.6 MiB/s, 137 objects/s recovering 2024-03-26T21:45:13.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:13.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:13.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:13.756 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:13.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:13.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:13.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:14.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:14.038 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:14.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:14.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:14.304 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:14.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:14.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:14.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:14.568 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:14.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:14.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:14.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:14.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:14.981 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:14.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:15.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:15.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.251 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:15.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:15 smithi055 ceph-mon[40587]: pgmap v772: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 983 op/s; 15274/311883 objects degraded (4.897%); 12109/311883 objects misplaced (3.883%); 20 MiB/s, 314 objects/s recovering 2024-03-26T21:45:15.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:15 smithi055 ceph-mon[29406]: pgmap v772: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 983 op/s; 15274/311883 objects degraded (4.897%); 12109/311883 objects misplaced (3.883%); 20 MiB/s, 314 objects/s recovering 2024-03-26T21:45:15.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:15.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:15 smithi067 ceph-mon[33346]: pgmap v772: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 983 op/s; 15274/311883 objects degraded (4.897%); 12109/311883 objects misplaced (3.883%); 20 MiB/s, 314 objects/s recovering 2024-03-26T21:45:15.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.529 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:15.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:15.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:15.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:15.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:15.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:16.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.081 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:16.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:16.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:16.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:16.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:16.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:16.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:16.504 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:16.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:16.640 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:16.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.779 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:16.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:16.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:16.917 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:16.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:17.007 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:45:17.008 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:17.008 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:17.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:17.120 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:17.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:17 smithi055 ceph-mon[29406]: pgmap v773: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 774 op/s; 15274/312000 objects degraded (4.896%); 12109/312000 objects misplaced (3.881%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:45:17.196 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:17 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 15274/312000 objects degraded (4.896%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:17.196 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:17 smithi055 ceph-mon[40587]: pgmap v773: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 774 op/s; 15274/312000 objects degraded (4.896%); 12109/312000 objects misplaced (3.881%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:45:17.196 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:17 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 15274/312000 objects degraded (4.896%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:17.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:17.255 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:17.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:17.407 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:17.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:17 smithi067 ceph-mon[33346]: pgmap v773: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 774 op/s; 15274/312000 objects degraded (4.896%); 12109/312000 objects misplaced (3.881%); 16 MiB/s, 263 objects/s recovering 2024-03-26T21:45:17.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:17 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 15274/312000 objects degraded (4.896%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:17.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:17.552 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:17.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:17.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:17.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:17.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:17.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:18.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:18.093 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:18.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:18.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:18.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:18.358 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:18.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:18.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:18.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:18.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:18.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:18.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:18.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:18.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:18.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:18.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:18.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:18.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:19.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:19.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:19.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.185 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:19.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:19.277 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:19.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:19.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:19.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:19.455 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:19.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:19.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:19 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:19.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:19 smithi055 ceph-mon[40587]: pgmap v774: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 843 op/s; 15274/302496 objects degraded (5.049%); 10670/302496 objects misplaced (3.527%); 19 MiB/s, 302 objects/s recovering 2024-03-26T21:45:19.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:19 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:19.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:19 smithi055 ceph-mon[29406]: pgmap v774: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 843 op/s; 15274/302496 objects degraded (5.049%); 10670/302496 objects misplaced (3.527%); 19 MiB/s, 302 objects/s recovering 2024-03-26T21:45:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:19.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.719 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:19.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:19 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:19.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:19 smithi067 ceph-mon[33346]: pgmap v774: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 843 op/s; 15274/302496 objects degraded (5.049%); 10670/302496 objects misplaced (3.527%); 19 MiB/s, 302 objects/s recovering 2024-03-26T21:45:19.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:19.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:19.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:19.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:20.076 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:20.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:20.119 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:20.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:20.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:20.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:20.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:20.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:20.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:20 smithi067 ceph-mon[33346]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:45:20.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:20 smithi067 ceph-mon[33346]: pgmap v776: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 837 op/s; 15274/298785 objects degraded (5.112%); 10169/298785 objects misplaced (3.403%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:20.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:20.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:20 smithi055 ceph-mon[29406]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:45:20.796 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:20 smithi055 ceph-mon[29406]: pgmap v776: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 837 op/s; 15274/298785 objects degraded (5.112%); 10169/298785 objects misplaced (3.403%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:20.796 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:20 smithi055 ceph-mon[40587]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T21:45:20.797 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:20 smithi055 ceph-mon[40587]: pgmap v776: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 23 MiB/s wr, 837 op/s; 15274/298785 objects degraded (5.112%); 10169/298785 objects misplaced (3.403%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:20.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:20.911 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:20.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:21.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:21.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:21.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:21.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:21.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:21.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:21.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:21.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:21.582 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:21.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:21.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:21 smithi067 ceph-mon[33346]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:45:21.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:21 smithi055 ceph-mon[29406]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:45:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:21 smithi055 ceph-mon[40587]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T21:45:21.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:21.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:21.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:21.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:22.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.131 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:22.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:22.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:22.365 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:22.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:22.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:22.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:22 smithi067 ceph-mon[33346]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:45:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:22 smithi067 ceph-mon[33346]: pgmap v779: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 736 KiB/s wr, 862 op/s; 15274/296916 objects degraded (5.144%); 10169/296916 objects misplaced (3.425%); 17 MiB/s, 274 objects/s recovering 2024-03-26T21:45:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 15274/298785 objects degraded (5.112%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:22.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:22 smithi067 ceph-mon[33346]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:45:22.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:22.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:22.802 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[40587]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:45:22.802 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[40587]: pgmap v779: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 736 KiB/s wr, 862 op/s; 15274/296916 objects degraded (5.144%); 10169/296916 objects misplaced (3.425%); 17 MiB/s, 274 objects/s recovering 2024-03-26T21:45:22.802 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 15274/298785 objects degraded (5.112%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:22.802 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[40587]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:45:22.805 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[29406]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T21:45:22.805 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[29406]: pgmap v779: 158 pgs: 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 736 KiB/s wr, 862 op/s; 15274/296916 objects degraded (5.144%); 10169/296916 objects misplaced (3.425%); 17 MiB/s, 274 objects/s recovering 2024-03-26T21:45:22.805 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 15274/298785 objects degraded (5.112%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:22.805 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:22 smithi055 ceph-mon[29406]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T21:45:22.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:22.930 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:22.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:23.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:23.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:23.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:23.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:23.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:23.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:23.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:23.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:23.691 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:45:23.692 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/590858837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:23.692 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:23 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:23.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/590858837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:23.692 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:23 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:23.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:23.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:23.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:23.886 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:23.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:23.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/590858837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:23 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:45:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:24.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:24.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:24.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.296 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:24.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:24.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:24.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:24.564 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:24.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:24.704 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:24.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:24.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[29406]: pgmap v781: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 875 op/s; 10110/287631 objects degraded (3.515%); 13775/287631 objects misplaced (4.789%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:24.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:24.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:45:24.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[29406]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:45:24.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[40587]: pgmap v781: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 875 op/s; 10110/287631 objects degraded (3.515%); 13775/287631 objects misplaced (4.789%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:24.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:24.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:45:24.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:24 smithi055 ceph-mon[40587]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:45:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:24.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:24.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:24.986 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:24.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:24 smithi067 ceph-mon[33346]: pgmap v781: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 875 op/s; 10110/287631 objects degraded (3.515%); 13775/287631 objects misplaced (4.789%); 20 MiB/s, 312 objects/s recovering 2024-03-26T21:45:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:24 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:24 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:45:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:24 smithi067 ceph-mon[33346]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T21:45:25.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:25.124 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:25.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:25.257 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:25.359 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.416 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:25.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:25.417 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:45:25.417 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:45:25.417 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:45:25.417 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:45:25.417 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:45:25.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:25.561 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:25.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.725 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:25.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:25.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:25.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:25.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:25.998 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:25.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:26.100 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:26.143 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:26.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:26.290 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:26.320 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:45:26.320 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "" 2024-03-26T21:45:26.320 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:45:26.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:26.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.606 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:26.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:26.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:45:26.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:26.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:26.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:26.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[29406]: pgmap v783: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 757 op/s; 7553/286881 objects degraded (2.633%); 16151/286881 objects misplaced (5.630%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[29406]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:26.923 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:26.924 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[40587]: pgmap v783: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 757 op/s; 7553/286881 objects degraded (2.633%); 16151/286881 objects misplaced (5.630%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:45:26.924 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:26 smithi055 ceph-mon[40587]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:45:26.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:26.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:45:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:26 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:26 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:26 smithi067 ceph-mon[33346]: pgmap v783: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 975 B/s rd, 0 B/s wr, 757 op/s; 7553/286881 objects degraded (2.633%); 16151/286881 objects misplaced (5.630%); 18 MiB/s, 287 objects/s recovering 2024-03-26T21:45:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:26 smithi067 ceph-mon[33346]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T21:45:27.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:27.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:27.156 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:27.258 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:27.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:27.474 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:27.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:27.608 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:27.636 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:27.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:27.696 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 7553/286881 objects degraded (2.633%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:27.696 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1894536169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:27.698 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 7553/286881 objects degraded (2.633%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:27.698 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1894536169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:27.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:27.715 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:45:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:27.753 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:45:27.794 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:45:27.836 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:45:27.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:27.879 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:45:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:27.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:27.923 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:45:27.959 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:45:27.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 7553/286881 objects degraded (2.633%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:27.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1894536169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:45:27.999 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:45:28.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:45:28.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:45:28.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:45:28.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:45:28.004 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:45:28.005 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:45:28.005 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:45:28.005 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:45:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:28.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:28.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:28.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:28.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:28.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:28.468 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:28.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:28.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:28.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:28.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:28.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:28.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:28.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:29.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:29.085 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:29.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:29.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[29406]: pgmap v785: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 7553/277074 objects degraded (2.726%); 15205/277074 objects misplaced (5.488%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:45:29.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:29.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:29.120 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:29.121 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:29.121 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[40587]: pgmap v785: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 7553/277074 objects degraded (2.726%); 15205/277074 objects misplaced (5.488%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:45:29.121 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:29.121 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:29.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:29.222 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:28 smithi067 ceph-mon[33346]: pgmap v785: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 7553/277074 objects degraded (2.726%); 15205/277074 objects misplaced (5.488%); 20 MiB/s, 316 objects/s recovering 2024-03-26T21:45:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:29.324 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:29.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:29.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.499 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:29.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:29.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:29.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:29.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:29.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:29.889 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:29.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:29.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:30.028 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:30.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:30.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:29 smithi067 ceph-mon[33346]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:45:30.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:30.292 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:30.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:29 smithi055 ceph-mon[40587]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:45:30.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:29 smithi055 ceph-mon[29406]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T21:45:30.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:30.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:30.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:30.662 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:30.694 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:30.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:30.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:30.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:30.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:45:30.952 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:30.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:30.959 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:30 smithi055 ceph-mon[40587]: pgmap v787: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 786 op/s; 7553/273462 objects degraded (2.762%); 14982/273462 objects misplaced (5.479%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:45:30.959 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:30 smithi055 ceph-mon[29406]: pgmap v787: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 786 op/s; 7553/273462 objects degraded (2.762%); 14982/273462 objects misplaced (5.479%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:45:31.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:31.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:31.183 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:31.219 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:30 smithi067 ceph-mon[33346]: pgmap v787: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 786 op/s; 7553/273462 objects degraded (2.762%); 14982/273462 objects misplaced (5.479%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:45:31.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.353 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:31.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:31.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:31.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:31.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:31.615 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:31.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:31.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:31.883 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:31.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:31.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:32.017 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:32.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:32.128 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:31 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 7553/273462 objects degraded (2.762%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:32.128 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:31 smithi055 ceph-mon[29406]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:45:32.128 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:31 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 7553/273462 objects degraded (2.762%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:32.128 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:31 smithi055 ceph-mon[40587]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:45:32.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:31 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 7553/273462 objects degraded (2.762%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:31 smithi067 ceph-mon[33346]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T21:45:32.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:32.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:32.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:32.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:32.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:32.568 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:32.670 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.708 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:32.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:32.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:32.839 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:32.901 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:45:32.927 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:45:32.928 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:45:32.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:32.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[40587]: pgmap v788: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 671 op/s; 7461/271566 objects degraded (2.747%); 14982/271566 objects misplaced (5.517%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T21:45:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:33.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:45:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[29406]: pgmap v788: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 671 op/s; 7461/271566 objects degraded (2.747%); 14982/271566 objects misplaced (5.517%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T21:45:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:45:33.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:33.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:33.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:33.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:32 smithi067 ceph-mon[33346]: pgmap v788: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 671 op/s; 7461/271566 objects degraded (2.747%); 14982/271566 objects misplaced (5.517%); 6.6 MiB/s, 105 objects/s recovering 2024-03-26T21:45:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:45:33.275 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:45:33.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:45:33.276 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-26T21:45:33.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:45:33.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:33.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:33.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.517 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:33.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:33.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:33.653 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:33.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:33.783 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:33.885 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:33.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:33.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:34.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:34.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:34.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:33 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:33 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:45:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:33 smithi067 ceph-mon[33346]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:45:34.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:34.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:45:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[40587]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:45:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:45:34.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2621453236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:45:34.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:33 smithi055 ceph-mon[29406]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T21:45:34.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:34.448 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:34.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:34.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:34.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:34.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.839 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:34.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:34.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:34.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:34.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:35.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:34 smithi055 ceph-mon[40587]: pgmap v791: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 729 op/s; 7461/263967 objects degraded (2.826%); 13718/263967 objects misplaced (5.197%); 9.3 MiB/s, 149 objects/s recovering 2024-03-26T21:45:35.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:34 smithi055 ceph-mon[29406]: pgmap v791: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 729 op/s; 7461/263967 objects degraded (2.826%); 13718/263967 objects misplaced (5.197%); 9.3 MiB/s, 149 objects/s recovering 2024-03-26T21:45:35.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:35.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:34 smithi067 ceph-mon[33346]: pgmap v791: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 729 op/s; 7461/263967 objects degraded (2.826%); 13718/263967 objects misplaced (5.197%); 9.3 MiB/s, 149 objects/s recovering 2024-03-26T21:45:35.241 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:35.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:35.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:35.504 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:35.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.642 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:35.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:35.743 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:35.779 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:35.881 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:35.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:35.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:36.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:36.052 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:36.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:35 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:45:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:35 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:45:36.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:35 smithi067 ceph-mon[33346]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:45:36.279 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:36.315 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[40587]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:45:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:35 smithi055 ceph-mon[29406]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T21:45:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:36.407 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:36.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:36.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:45:36.448 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:36.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:45:36.550 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:36.615 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:36.717 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:36.773 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:36.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:36.911 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:36.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:37.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:37.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:37.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:36 smithi067 ceph-mon[33346]: pgmap v793: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 559 op/s; 7461/263394 objects degraded (2.833%); 13346/263394 objects misplaced (5.067%); 11 MiB/s, 174 objects/s recovering 2024-03-26T21:45:37.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:36 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 7461/263394 objects degraded (2.833%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:37.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:37.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:36 smithi055 ceph-mon[40587]: pgmap v793: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 559 op/s; 7461/263394 objects degraded (2.833%); 13346/263394 objects misplaced (5.067%); 11 MiB/s, 174 objects/s recovering 2024-03-26T21:45:37.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:36 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 7461/263394 objects degraded (2.833%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:36 smithi055 ceph-mon[29406]: pgmap v793: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 559 op/s; 7461/263394 objects degraded (2.833%); 13346/263394 objects misplaced (5.067%); 11 MiB/s, 174 objects/s recovering 2024-03-26T21:45:37.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:36 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 7461/263394 objects degraded (2.833%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:37.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:37.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:37.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:37.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:37.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:37.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:37.835 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:37.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:37.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:38.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:37 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:45:38.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:37 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:45:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:37 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:45:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:37 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:45:38.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:38.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:37 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:37 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:45:38.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:38.383 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:38.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:38.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:38.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:38.642 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:38.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:38.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:38.875 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:38.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:38.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:39.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:39.038 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:39.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:39.167 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: pgmap v794: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 846 op/s; 7302/256335 objects degraded (2.849%); 12002/256335 objects misplaced (4.682%); 18 MiB/s, 293 objects/s recovering 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:38 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:39.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:45:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:45:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: pgmap v794: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 846 op/s; 7302/256335 objects degraded (2.849%); 12002/256335 objects misplaced (4.682%); 18 MiB/s, 293 objects/s recovering 2024-03-26T21:45:39.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: pgmap v794: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 846 op/s; 7302/256335 objects degraded (2.849%); 12002/256335 objects misplaced (4.682%); 18 MiB/s, 293 objects/s recovering 2024-03-26T21:45:39.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:45:39.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:38 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:39.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:39.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:39.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:39.662 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:39.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:39.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:39.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:39.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:39.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:40.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.091 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:40.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:40.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:40.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:39 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:45:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:39 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:45:40.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:39 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:39 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:45:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:45:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:45:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:45:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:39 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:40.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:40.381 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:40.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:40.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:40.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:40.642 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:40.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:40.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:40.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:40.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:41.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:41.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:41.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:41.165 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:41.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:40 smithi067 ceph-mon[33346]: pgmap v795: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 585 op/s; 7302/254121 objects degraded (2.873%); 11565/254121 objects misplaced (4.551%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:41.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:41.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:41.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:40 smithi055 ceph-mon[40587]: pgmap v795: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 585 op/s; 7302/254121 objects degraded (2.873%); 11565/254121 objects misplaced (4.551%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:40 smithi055 ceph-mon[29406]: pgmap v795: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 585 op/s; 7302/254121 objects degraded (2.873%); 11565/254121 objects misplaced (4.551%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:41.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:41.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:41.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.574 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:41.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:41.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:41.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:41.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:41.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:41.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:42.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:42.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:42.231 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:42.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 7302/254121 objects degraded (2.873%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:42.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 7302/254121 objects degraded (2.873%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:42.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:42.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:42.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 7302/254121 objects degraded (2.873%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:42.496 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:42.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:42.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:42.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:42.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:42.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:42.885 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:42.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:42.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:43.013 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:43.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:43.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:43.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:43.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:43.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:43 smithi055 ceph-mon[40587]: pgmap v796: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 444 op/s; 7138/252858 objects degraded (2.823%); 11565/252858 objects misplaced (4.574%); 12 MiB/s, 188 objects/s recovering 2024-03-26T21:45:43.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:43 smithi055 ceph-mon[29406]: pgmap v796: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 444 op/s; 7138/252858 objects degraded (2.823%); 11565/252858 objects misplaced (4.574%); 12 MiB/s, 188 objects/s recovering 2024-03-26T21:45:43.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:43 smithi067 ceph-mon[33346]: pgmap v796: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 444 op/s; 7138/252858 objects degraded (2.823%); 11565/252858 objects misplaced (4.574%); 12 MiB/s, 188 objects/s recovering 2024-03-26T21:45:43.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:43.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:43.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.674 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:43.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:43.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:43.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:43.881 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:43.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:43.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:44.049 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.089 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:44.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:44.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:44.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:44.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:44.357 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:44.459 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.497 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:44.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:44.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:44.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:44.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:44.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:44.863 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:44.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:44.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:44.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:45.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:45.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:45.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:45.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[40587]: pgmap v797: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 7138/246447 objects degraded (2.896%); 10272/246447 objects misplaced (4.168%); 15 MiB/s, 232 objects/s recovering 2024-03-26T21:45:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[29406]: pgmap v797: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 7138/246447 objects degraded (2.896%); 10272/246447 objects misplaced (4.168%); 15 MiB/s, 232 objects/s recovering 2024-03-26T21:45:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:45 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:45.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:45 smithi067 ceph-mon[33346]: pgmap v797: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 7138/246447 objects degraded (2.896%); 10272/246447 objects misplaced (4.168%); 15 MiB/s, 232 objects/s recovering 2024-03-26T21:45:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:45 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:45:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:45 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:45:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:45 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:45:45.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:45.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:45.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:45.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:45.940 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:45.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:46.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:46.072 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:46.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:46.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:46.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:46.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:46.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:46.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:46.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.614 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:46.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:46.715 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:46.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:46.853 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:46.889 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:46.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:46.990 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:47.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:47.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.161 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:47.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:47.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:47.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:47 smithi055 ceph-mon[40587]: pgmap v798: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 541 op/s; 7138/246009 objects degraded (2.902%); 9770/246009 objects misplaced (3.971%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:45:47.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 7138/246009 objects degraded (2.902%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:47 smithi055 ceph-mon[29406]: pgmap v798: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 541 op/s; 7138/246009 objects degraded (2.902%); 9770/246009 objects misplaced (3.971%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:45:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 7138/246009 objects degraded (2.902%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:47.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:47.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:47 smithi067 ceph-mon[33346]: pgmap v798: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 541 op/s; 7138/246009 objects degraded (2.902%); 9770/246009 objects misplaced (3.971%); 14 MiB/s, 228 objects/s recovering 2024-03-26T21:45:47.442 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 7138/246009 objects degraded (2.902%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:47.516 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:47.550 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:47.652 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:47.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:47.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:47.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:47.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:47.949 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:47.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:48.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:48.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:48.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:48.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:48.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:45:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:45:48.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:45:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:45:48.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:48.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:48.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:48 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:45:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:48 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:45:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:45:48.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.613 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:48.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:48.714 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:48.744 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:48.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:48.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:48.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:48.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:49.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:49.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:49.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:49.243 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:49.244 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:49.244 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:49 smithi055 ceph-mon[40587]: pgmap v799: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 6965/239565 objects degraded (2.907%); 8203/239565 objects misplaced (3.424%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:49.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:49.244 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:49 smithi055 ceph-mon[29406]: pgmap v799: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 6965/239565 objects degraded (2.907%); 8203/239565 objects misplaced (3.424%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:49.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:49.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:49.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:49 smithi067 ceph-mon[33346]: pgmap v799: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 662 op/s; 6965/239565 objects degraded (2.907%); 8203/239565 objects misplaced (3.424%); 18 MiB/s, 281 objects/s recovering 2024-03-26T21:45:49.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:49.535 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:49.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:49.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:49.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:49.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:49.922 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:49.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:50.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:50.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:50.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:50.325 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:50.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:50.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:50.583 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:50.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:50.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.845 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:50.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:50.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:50.973 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:50.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:51.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:51.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:51.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:51.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:51.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:45:51.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:45:51.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[40587]: pgmap v800: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 6965/237333 objects degraded (2.935%); 7495/237333 objects misplaced (3.158%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[29406]: pgmap v800: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 6965/237333 objects degraded (2.935%); 7495/237333 objects misplaced (3.158%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:45:51.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:51 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:45:51.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:51.365 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:51.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:51 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:45:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:51 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:45:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:51 smithi067 ceph-mon[33346]: pgmap v800: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 528 op/s; 6965/237333 objects degraded (2.935%); 7495/237333 objects misplaced (3.158%); 17 MiB/s, 277 objects/s recovering 2024-03-26T21:45:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:51 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:45:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:51 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:45:51.491 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:51.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.617 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:51.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:51.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:51.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:51.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:51.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:51.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:52.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:52.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:52.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:52.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:52.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:52.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[29406]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:45:52.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6965/237333 objects degraded (2.935%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:52.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[29406]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:45:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[40587]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:45:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6965/237333 objects degraded (2.935%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:52 smithi055 ceph-mon[40587]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:45:52.386 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.425 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:52.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:52 smithi067 ceph-mon[33346]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T21:45:52.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6965/237333 objects degraded (2.935%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T21:45:52.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:52 smithi067 ceph-mon[33346]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T21:45:52.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:52.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:52.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:52.699 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:52.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:52.836 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:52.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:52.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:52.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:53.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:53.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:53.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:53.226 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[29406]: pgmap v802: 158 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 3967/235941 objects degraded (1.681%); 10323/235941 objects misplaced (4.375%); 18 MiB/s, 290 objects/s recovering 2024-03-26T21:45:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:45:53.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:45:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[40587]: pgmap v802: 158 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 3967/235941 objects degraded (1.681%); 10323/235941 objects misplaced (4.375%); 18 MiB/s, 290 objects/s recovering 2024-03-26T21:45:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:45:53.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:53 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:45:53.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:53.352 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:53.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:53.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:53 smithi067 ceph-mon[33346]: pgmap v802: 158 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 3967/235941 objects degraded (1.681%); 10323/235941 objects misplaced (4.375%); 18 MiB/s, 290 objects/s recovering 2024-03-26T21:45:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:53 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:45:53.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:53 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:45:53.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:53.611 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:53.713 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.743 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:53.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:53.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:53.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:53.999 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:53.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:54.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:54.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:54.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:54 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:45:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:54 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:45:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:54 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:45:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:54 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:45:54.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:54.544 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:45:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:45:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:45:54.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:45:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:45:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:45:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:45:54.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:54 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:45:54.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:54.684 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:54.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:54.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:54.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:54.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:55.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:55.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:55.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.211 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:55.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:55.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:55.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[40587]: pgmap v804: 158 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 678 op/s; 1738/229752 objects degraded (0.756%); 10801/229752 objects misplaced (4.701%); 21 MiB/s, 357 objects/s recovering 2024-03-26T21:45:55.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[40587]: 10.11 scrub starts 2024-03-26T21:45:55.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:45:55.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:45:55.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[40587]: 10.11 scrub ok 2024-03-26T21:45:55.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[29406]: pgmap v804: 158 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 678 op/s; 1738/229752 objects degraded (0.756%); 10801/229752 objects misplaced (4.701%); 21 MiB/s, 357 objects/s recovering 2024-03-26T21:45:55.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[29406]: 10.11 scrub starts 2024-03-26T21:45:55.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:45:55.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:45:55.445 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:55 smithi055 ceph-mon[29406]: 10.11 scrub ok 2024-03-26T21:45:55.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:55 smithi067 ceph-mon[33346]: pgmap v804: 158 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 678 op/s; 1738/229752 objects degraded (0.756%); 10801/229752 objects misplaced (4.701%); 21 MiB/s, 357 objects/s recovering 2024-03-26T21:45:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:55 smithi067 ceph-mon[33346]: 10.11 scrub starts 2024-03-26T21:45:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:55 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:45:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:55 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:45:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:55 smithi067 ceph-mon[33346]: 10.11 scrub ok 2024-03-26T21:45:55.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:55.603 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:55.705 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:55.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:55.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:55.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:55.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:55.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:56.005 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:56.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:56.138 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:56.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:56.242 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:45:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:56.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:56.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:56.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:56 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:45:56.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:56 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:45:56.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:56.532 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:56.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:56 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:45:56.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:56 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:45:56.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:56 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:45:56.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:56 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:45:56.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:56.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:56.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:56.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:56.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:57.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:45:57.057 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:45:57.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:57.190 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:57.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:57.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: pgmap v805: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 430 op/s; 1530/229263 objects degraded (0.667%); 10667/229263 objects misplaced (4.653%); 14 MiB/s, 283 objects/s recovering 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1530/229263 objects degraded (0.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: pgmap v805: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 430 op/s; 1530/229263 objects degraded (0.667%); 10667/229263 objects misplaced (4.653%); 14 MiB/s, 283 objects/s recovering 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:45:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1530/229263 objects degraded (0.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:45:57.320 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:57.422 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:57.451 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:45:57.452 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:45:57.452 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: pgmap v805: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 430 op/s; 1530/229263 objects degraded (0.667%); 10667/229263 objects misplaced (4.653%); 14 MiB/s, 283 objects/s recovering 2024-03-26T21:45:57.452 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:45:57.452 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:45:57.452 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1530/229263 objects degraded (0.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:45:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.453 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:57.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:57.555 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:57.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:57.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:45:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:45:57.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:57.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:57.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:57.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:57.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:58.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.094 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:58.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:58.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:45:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:45:58.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:58.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:58 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:45:58.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:58 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:45:58.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:58 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:45:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:58 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:45:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:58 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:45:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:58 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:45:58.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:45:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:58.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:45:58.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:58.753 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:58.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:58.882 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:58.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:58.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:59.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:59.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:45:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.139 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:45:59.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:59.272 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:45:59 smithi055 ceph-mon[40587]: pgmap v806: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 118 op/s; 1530/222096 objects degraded (0.689%); 9169/222096 objects misplaced (4.128%); 5.3 MiB/s, 138 objects/s recovering 2024-03-26T21:45:59.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:45:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:45:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:45:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:45:59 smithi055 ceph-mon[29406]: pgmap v806: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 118 op/s; 1530/222096 objects degraded (0.689%); 9169/222096 objects misplaced (4.128%); 5.3 MiB/s, 138 objects/s recovering 2024-03-26T21:45:59.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:45:59.401 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:45:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:45:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:45:59 smithi067 ceph-mon[33346]: pgmap v806: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 118 op/s; 1530/222096 objects degraded (0.689%); 9169/222096 objects misplaced (4.128%); 5.3 MiB/s, 138 objects/s recovering 2024-03-26T21:45:59.502 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:45:59.537 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:45:59.639 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.669 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:45:59.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:45:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:45:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:45:59.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:45:59.931 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:45:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:45:59.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:00.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:00.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:00.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.205 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:00.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:00.307 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:00.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:00.436 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:00.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:46:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:00 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:46:00.522 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:46:00.523 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:46:00.524 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:46:00.524 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:00 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:46:00.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:46:00.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:46:00.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:00.591 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:46:00.610 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:00.629 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:46:00.666 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:46:00.696 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:46:00.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:00.727 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:46:00.742 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:00.762 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:46:00.765 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:46:00.766 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:46:00.766 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:46:00.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:00.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:00.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:01.003 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:01.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:01.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.266 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:01.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:01.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:01.406 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:01.477 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[40587]: pgmap v808: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 143 op/s; 1530/220104 objects degraded (0.695%); 9169/220104 objects misplaced (4.166%); 4.6 MiB/s, 127 objects/s recovering 2024-03-26T21:46:01.477 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[40587]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:46:01.477 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2103987366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:01.477 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[29406]: pgmap v808: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 143 op/s; 1530/220104 objects degraded (0.695%); 9169/220104 objects misplaced (4.166%); 4.6 MiB/s, 127 objects/s recovering 2024-03-26T21:46:01.478 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[29406]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:46:01.478 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2103987366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:01 smithi067 ceph-mon[33346]: pgmap v808: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 143 op/s; 1530/220104 objects degraded (0.695%); 9169/220104 objects misplaced (4.166%); 4.6 MiB/s, 127 objects/s recovering 2024-03-26T21:46:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:01 smithi067 ceph-mon[33346]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T21:46:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2103987366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:01.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:01.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:01.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:01.710 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:46:01.713 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:46:01.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:01.805 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:01.907 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:01.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:01.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:02.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:02.078 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:02.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:02.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:02.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:02.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:02.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:02.485 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:02.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1530/220104 objects degraded (0.695%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:02.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1530/220104 objects degraded (0.695%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:02.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1530/220104 objects degraded (0.695%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:02.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:02.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:02.752 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:02.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:02.886 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:02.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:02.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:03.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:03.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:03.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:03.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:03.288 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:03.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:03 smithi067 ceph-mon[33346]: pgmap v810: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1530/218829 objects degraded (0.699%); 8985/218829 objects misplaced (4.106%) 2024-03-26T21:46:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:03 smithi067 ceph-mon[33346]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:46:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:03 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:46:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:03 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:46:03.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[40587]: pgmap v810: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1530/218829 objects degraded (0.699%); 8985/218829 objects misplaced (4.106%) 2024-03-26T21:46:03.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[40587]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:46:03.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:03.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:46:03.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:46:03.498 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[29406]: pgmap v810: 158 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1530/218829 objects degraded (0.699%); 8985/218829 objects misplaced (4.106%) 2024-03-26T21:46:03.498 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[29406]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:46:03.498 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:03.498 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:46:03.498 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:03 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:46:03.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:03.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:03.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:03.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:03.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:03.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:03.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:03.950 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:03.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:04.051 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:04.083 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:04.159 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:04.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:04.202 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":810,"stamp":"2024-03-26T21:46:01.669696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4767816128,"num_objects":72943,"num_object_clones":0,"num_object_copies":218829,"num_objects_missing_on_primary":656,"num_objects_missing":656,"num_objects_degraded":1530,"num_objects_misplaced":8985,"num_objects_unfound":0,"num_objects_dirty":72943,"num_whiteouts":0,"num_read":2241,"num_read_kb":2089,"num_write":115538,"num_write_kb":5392990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30210,"num_bytes_recovered":1931878474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149026,"ondisk_log_size":149026,"up":474,"acting":471,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":484,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19618792,"kb_used_data":14918752,"kb_used_omap":20,"kb_used_meta":4699883,"kb_avail":730342424,"statfs":{"total":767960285184,"available":747870642176,"internally_reserved":0,"allocated":15276802048,"data_stored":15269896513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20884,"internal_metadata":4812680812},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-238583808,"num_objects":-3641,"num_object_clones":0,"num_object_copies":-10923,"num_objects_missing_on_primary":-179,"num_objects_missing":-179,"num_objects_degraded":-208,"num_objects_misplaced":-1816,"num_objects_unfound":0,"num_objects_dirty":-3641,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-498,"num_write_kb":-205664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-26,"num_bytes_recovered":-15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002752"},"pg_stats":[{"pgid":"10.16","version":"384'5695","reported_seq":10650,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.765259+0000","last_change":"2024-03-26T21:44:56.836539+0000","last_active":"2024-03-26T21:45:56.765259+0000","last_peered":"2024-03-26T21:45:56.765259+0000","last_clean":"2024-03-26T21:45:56.765259+0000","last_became_active":"2024-03-26T21:44:56.832240+0000","last_became_peered":"2024-03-26T21:44:56.832240+0000","last_unstale":"2024-03-26T21:45:56.765259+0000","last_undegraded":"2024-03-26T21:45:56.765259+0000","last_fullsized":"2024-03-26T21:45:56.765259+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:58.676918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146931712,"num_objects":2242,"num_object_clones":0,"num_object_copies":6726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4094,"num_write_kb":202752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1605,"num_bytes_recovered":105185280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":400,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131576+0000","last_change":"2024-03-26T21:41:48.911741+0000","last_active":"2024-03-26T21:46:00.131576+0000","last_peered":"2024-03-26T21:46:00.131576+0000","last_clean":"2024-03-26T21:46:00.131576+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:46:00.131576+0000","last_undegraded":"2024-03-26T21:46:00.131576+0000","last_fullsized":"2024-03-26T21:46:00.131576+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:48.911605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:41:48.911605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":709,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.840491+0000","last_change":"2024-03-26T21:43:46.178825+0000","last_active":"2024-03-26T21:45:51.840491+0000","last_peered":"2024-03-26T21:45:51.840491+0000","last_clean":"2024-03-26T21:45:51.840491+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:45:51.840491+0000","last_undegraded":"2024-03-26T21:45:51.840491+0000","last_fullsized":"2024-03-26T21:45:51.840491+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:46.178692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:46.178692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":693,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841415+0000","last_change":"2024-03-26T21:40:20.651612+0000","last_active":"2024-03-26T21:45:51.841415+0000","last_peered":"2024-03-26T21:45:51.841415+0000","last_clean":"2024-03-26T21:45:51.841415+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:45:51.841415+0000","last_undegraded":"2024-03-26T21:45:51.841415+0000","last_fullsized":"2024-03-26T21:45:51.841415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:20.651494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:40:20.651494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":435,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131307+0000","last_change":"2024-03-26T21:43:49.515729+0000","last_active":"2024-03-26T21:46:00.131307+0000","last_peered":"2024-03-26T21:46:00.131307+0000","last_clean":"2024-03-26T21:46:00.131307+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:46:00.131307+0000","last_undegraded":"2024-03-26T21:46:00.131307+0000","last_fullsized":"2024-03-26T21:46:00.131307+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:43:49.515628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:49.515628+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"386'5819","reported_seq":8870,"reported_epoch":386,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:46:01.030108+0000","last_change":"2024-03-26T21:45:50.365910+0000","last_active":"2024-03-26T21:46:01.030108+0000","last_peered":"2024-03-26T21:46:01.030108+0000","last_clean":"2024-03-26T21:44:06.293234+0000","last_became_active":"2024-03-26T21:44:55.649185+0000","last_became_peered":"2024-03-26T21:44:55.649185+0000","last_unstale":"2024-03-26T21:46:01.030108+0000","last_undegraded":"2024-03-26T21:44:53.322095+0000","last_fullsized":"2024-03-26T21:44:53.318652+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":656,"num_objects_missing":656,"num_objects_degraded":418,"num_objects_misplaced":2477,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4206,"num_write_kb":207424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":967,"num_bytes_recovered":63373312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,6],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"1,2,4,6","objects":1152},{"shards":"2,4,5,6","objects":13},{"shards":"2,5,6","objects":656},{"shards":"4,6","objects":37}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845407+0000","last_change":"2024-03-26T21:44:49.396759+0000","last_active":"2024-03-26T21:45:51.845407+0000","last_peered":"2024-03-26T21:45:51.845407+0000","last_clean":"2024-03-26T21:45:51.845407+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:45:51.845407+0000","last_undegraded":"2024-03-26T21:45:51.845407+0000","last_fullsized":"2024-03-26T21:45:51.845407+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:20.336684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:41:20.336684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":400,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131413+0000","last_change":"2024-03-26T21:41:38.083671+0000","last_active":"2024-03-26T21:46:00.131413+0000","last_peered":"2024-03-26T21:46:00.131413+0000","last_clean":"2024-03-26T21:46:00.131413+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:46:00.131413+0000","last_undegraded":"2024-03-26T21:46:00.131413+0000","last_fullsized":"2024-03-26T21:46:00.131413+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:41:38.083526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846433+0000","last_change":"2024-03-26T21:44:49.408197+0000","last_active":"2024-03-26T21:45:51.846433+0000","last_peered":"2024-03-26T21:45:51.846433+0000","last_clean":"2024-03-26T21:45:51.846433+0000","last_became_active":"2024-03-26T21:44:49.404120+0000","last_became_peered":"2024-03-26T21:44:49.404120+0000","last_unstale":"2024-03-26T21:45:51.846433+0000","last_undegraded":"2024-03-26T21:45:51.846433+0000","last_fullsized":"2024-03-26T21:45:51.846433+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:57.090868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.090868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":67,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845476+0000","last_change":"2024-03-26T21:44:49.489035+0000","last_active":"2024-03-26T21:45:51.845476+0000","last_peered":"2024-03-26T21:45:51.845476+0000","last_clean":"2024-03-26T21:45:51.845476+0000","last_became_active":"2024-03-26T21:44:49.407818+0000","last_became_peered":"2024-03-26T21:44:49.407818+0000","last_unstale":"2024-03-26T21:45:51.845476+0000","last_undegraded":"2024-03-26T21:45:51.845476+0000","last_fullsized":"2024-03-26T21:45:51.845476+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:44:00.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:44:00.432782+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"384'5811","reported_seq":9301,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.624746+0000","last_change":"2024-03-26T21:44:33.841715+0000","last_active":"2024-03-26T21:45:57.624746+0000","last_peered":"2024-03-26T21:45:57.624746+0000","last_clean":"2024-03-26T21:45:57.624746+0000","last_became_active":"2024-03-26T21:44:33.836990+0000","last_became_peered":"2024-03-26T21:44:33.836990+0000","last_unstale":"2024-03-26T21:45:57.624746+0000","last_undegraded":"2024-03-26T21:45:57.624746+0000","last_fullsized":"2024-03-26T21:45:57.624746+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:00.474049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148668416,"num_objects":2269,"num_object_clones":0,"num_object_copies":6807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4203,"num_write_kb":207072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1676,"num_bytes_recovered":109838336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":689,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841333+0000","last_change":"2024-03-26T21:44:04.302643+0000","last_active":"2024-03-26T21:45:51.841333+0000","last_peered":"2024-03-26T21:45:51.841333+0000","last_clean":"2024-03-26T21:45:51.841333+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:45:51.841333+0000","last_undegraded":"2024-03-26T21:45:51.841333+0000","last_fullsized":"2024-03-26T21:45:51.841333+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:04.302332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:44:04.302332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.849613+0000","last_change":"2024-03-26T21:43:44.320137+0000","last_active":"2024-03-26T21:45:51.849613+0000","last_peered":"2024-03-26T21:45:51.849613+0000","last_clean":"2024-03-26T21:45:51.849613+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:45:51.849613+0000","last_undegraded":"2024-03-26T21:45:51.849613+0000","last_fullsized":"2024-03-26T21:45:51.849613+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:44.319965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:43:44.319965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":719,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850430+0000","last_change":"2024-03-26T21:43:56.501311+0000","last_active":"2024-03-26T21:45:51.850430+0000","last_peered":"2024-03-26T21:45:51.850430+0000","last_clean":"2024-03-26T21:45:51.850430+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:45:51.850430+0000","last_undegraded":"2024-03-26T21:45:51.850430+0000","last_fullsized":"2024-03-26T21:45:51.850430+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.501111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.501111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":459,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131675+0000","last_change":"2024-03-26T21:44:49.460726+0000","last_active":"2024-03-26T21:46:00.131675+0000","last_peered":"2024-03-26T21:46:00.131675+0000","last_clean":"2024-03-26T21:46:00.131675+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:46:00.131675+0000","last_undegraded":"2024-03-26T21:46:00.131675+0000","last_fullsized":"2024-03-26T21:46:00.131675+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:44:07.631570+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:07.631570+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"384'5798","reported_seq":16738,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.786494+0000","last_change":"2024-03-26T21:45:21.864948+0000","last_active":"2024-03-26T21:45:57.786494+0000","last_peered":"2024-03-26T21:45:57.786494+0000","last_clean":"2024-03-26T21:45:57.786494+0000","last_became_active":"2024-03-26T21:45:21.852312+0000","last_became_peered":"2024-03-26T21:45:21.852312+0000","last_unstale":"2024-03-26T21:45:57.786494+0000","last_undegraded":"2024-03-26T21:45:57.786494+0000","last_fullsized":"2024-03-26T21:45:57.786494+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:01.239964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139362304,"num_objects":2127,"num_object_clones":0,"num_object_copies":6381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":202400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4536,"num_bytes_recovered":297271296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850642+0000","last_change":"2024-03-26T21:44:49.335153+0000","last_active":"2024-03-26T21:45:51.850642+0000","last_peered":"2024-03-26T21:45:51.850642+0000","last_clean":"2024-03-26T21:45:51.850642+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:45:51.850642+0000","last_undegraded":"2024-03-26T21:45:51.850642+0000","last_fullsized":"2024-03-26T21:45:51.850642+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:48.999826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.999826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":367,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:52.591589+0000","last_change":"2024-03-26T21:45:52.591636+0000","last_active":"2024-03-26T21:45:52.591589+0000","last_peered":"2024-03-26T21:45:52.591589+0000","last_clean":"2024-03-26T21:45:52.591589+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:45:52.591589+0000","last_undegraded":"2024-03-26T21:45:52.591589+0000","last_fullsized":"2024-03-26T21:45:52.591589+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:52.591442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:45:52.591442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:02.726093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":719,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.849943+0000","last_change":"2024-03-26T21:43:48.314549+0000","last_active":"2024-03-26T21:45:51.849943+0000","last_peered":"2024-03-26T21:45:51.849943+0000","last_clean":"2024-03-26T21:45:51.849943+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:45:51.849943+0000","last_undegraded":"2024-03-26T21:45:51.849943+0000","last_fullsized":"2024-03-26T21:45:51.849943+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:48.314399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:35.930502+0000","last_clean_scrub_stamp":"2024-03-26T21:43:48.314399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":433,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131880+0000","last_change":"2024-03-26T21:44:49.448622+0000","last_active":"2024-03-26T21:46:00.131880+0000","last_peered":"2024-03-26T21:46:00.131880+0000","last_clean":"2024-03-26T21:46:00.131880+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:46:00.131880+0000","last_undegraded":"2024-03-26T21:46:00.131880+0000","last_fullsized":"2024-03-26T21:46:00.131880+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:43:02.671224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:02.671224+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"384'5946","reported_seq":12796,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.794217+0000","last_change":"2024-03-26T21:44:50.329781+0000","last_active":"2024-03-26T21:45:56.794217+0000","last_peered":"2024-03-26T21:45:56.794217+0000","last_clean":"2024-03-26T21:45:56.794217+0000","last_became_active":"2024-03-26T21:44:50.327076+0000","last_became_peered":"2024-03-26T21:44:50.327076+0000","last_unstale":"2024-03-26T21:45:56.794217+0000","last_undegraded":"2024-03-26T21:45:56.794217+0000","last_fullsized":"2024-03-26T21:45:56.794217+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151617536,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4286,"num_write_kb":211168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":112197632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":401,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.132434+0000","last_change":"2024-03-26T21:43:53.584198+0000","last_active":"2024-03-26T21:46:00.132434+0000","last_peered":"2024-03-26T21:46:00.132434+0000","last_clean":"2024-03-26T21:46:00.132434+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:46:00.132434+0000","last_undegraded":"2024-03-26T21:46:00.132434+0000","last_fullsized":"2024-03-26T21:46:00.132434+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:53.584060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:53.584060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":712,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841468+0000","last_change":"2024-03-26T21:40:16.881228+0000","last_active":"2024-03-26T21:45:51.841468+0000","last_peered":"2024-03-26T21:45:51.841468+0000","last_clean":"2024-03-26T21:45:51.841468+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:45:51.841468+0000","last_undegraded":"2024-03-26T21:45:51.841468+0000","last_fullsized":"2024-03-26T21:45:51.841468+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:16.881096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.881096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":713,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842123+0000","last_change":"2024-03-26T21:42:56.212250+0000","last_active":"2024-03-26T21:45:51.842123+0000","last_peered":"2024-03-26T21:45:51.842123+0000","last_clean":"2024-03-26T21:45:51.842123+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:45:51.842123+0000","last_undegraded":"2024-03-26T21:45:51.842123+0000","last_fullsized":"2024-03-26T21:45:51.842123+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.212112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":798,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843218+0000","last_change":"2024-03-26T21:44:49.478438+0000","last_active":"2024-03-26T21:45:51.843218+0000","last_peered":"2024-03-26T21:45:51.843218+0000","last_clean":"2024-03-26T21:45:51.843218+0000","last_became_active":"2024-03-26T21:44:49.417143+0000","last_became_peered":"2024-03-26T21:44:49.417143+0000","last_unstale":"2024-03-26T21:45:51.843218+0000","last_undegraded":"2024-03-26T21:45:51.843218+0000","last_fullsized":"2024-03-26T21:45:51.843218+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:57.266450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.266450+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"384'5873","reported_seq":11192,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.757872+0000","last_change":"2024-03-26T21:44:42.840723+0000","last_active":"2024-03-26T21:45:57.757872+0000","last_peered":"2024-03-26T21:45:57.757872+0000","last_clean":"2024-03-26T21:45:57.757872+0000","last_became_active":"2024-03-26T21:44:42.836745+0000","last_became_peered":"2024-03-26T21:44:42.836745+0000","last_unstale":"2024-03-26T21:45:57.757872+0000","last_undegraded":"2024-03-26T21:45:57.757872+0000","last_fullsized":"2024-03-26T21:45:57.757872+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:45:59.512630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":209344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1727,"num_bytes_recovered":113180672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847052+0000","last_change":"2024-03-26T21:44:49.396102+0000","last_active":"2024-03-26T21:45:51.847052+0000","last_peered":"2024-03-26T21:45:51.847052+0000","last_clean":"2024-03-26T21:45:51.847052+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:45:51.847052+0000","last_undegraded":"2024-03-26T21:45:51.847052+0000","last_fullsized":"2024-03-26T21:45:51.847052+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.247230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.247230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":424,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131970+0000","last_change":"2024-03-26T21:44:49.396364+0000","last_active":"2024-03-26T21:46:00.131970+0000","last_peered":"2024-03-26T21:46:00.131970+0000","last_clean":"2024-03-26T21:46:00.131970+0000","last_became_active":"2024-03-26T21:44:49.395939+0000","last_became_peered":"2024-03-26T21:44:49.395939+0000","last_unstale":"2024-03-26T21:46:00.131970+0000","last_undegraded":"2024-03-26T21:46:00.131970+0000","last_fullsized":"2024-03-26T21:46:00.131970+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:06.630161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.630161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":400,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131538+0000","last_change":"2024-03-26T21:43:50.520924+0000","last_active":"2024-03-26T21:46:00.131538+0000","last_peered":"2024-03-26T21:46:00.131538+0000","last_clean":"2024-03-26T21:46:00.131538+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:46:00.131538+0000","last_undegraded":"2024-03-26T21:46:00.131538+0000","last_fullsized":"2024-03-26T21:46:00.131538+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:50.520786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:50.520786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"234'18","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846122+0000","last_change":"2024-03-26T21:44:49.505757+0000","last_active":"2024-03-26T21:45:51.846122+0000","last_peered":"2024-03-26T21:45:51.846122+0000","last_clean":"2024-03-26T21:45:51.846122+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:45:51.846122+0000","last_undegraded":"2024-03-26T21:45:51.846122+0000","last_fullsized":"2024-03-26T21:45:51.846122+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'18","last_scrub_stamp":"2024-03-26T21:40:38.654652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.654652+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"386'5859","reported_seq":9287,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.880680+0000","last_change":"2024-03-26T21:44:17.932725+0000","last_active":"2024-03-26T21:46:00.880680+0000","last_peered":"2024-03-26T21:46:00.880680+0000","last_clean":"2024-03-26T21:46:00.880680+0000","last_became_active":"2024-03-26T21:44:17.910864+0000","last_became_peered":"2024-03-26T21:44:17.910864+0000","last_unstale":"2024-03-26T21:46:00.880680+0000","last_undegraded":"2024-03-26T21:46:00.880680+0000","last_fullsized":"2024-03-26T21:46:00.880680+0000","mapping_epoch":353,"log_start":"336'37","ondisk_log_start":"336'37","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:03.420092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147587072,"num_objects":2252,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4258,"num_write_kb":208320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1642,"num_bytes_recovered":107610112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":623,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852597+0000","last_change":"2024-03-26T21:43:06.349697+0000","last_active":"2024-03-26T21:45:51.852597+0000","last_peered":"2024-03-26T21:45:51.852597+0000","last_clean":"2024-03-26T21:45:51.852597+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:45:51.852597+0000","last_undegraded":"2024-03-26T21:45:51.852597+0000","last_fullsized":"2024-03-26T21:45:51.852597+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:06.349429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.349429+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":964,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848902+0000","last_change":"2024-03-26T21:43:49.506804+0000","last_active":"2024-03-26T21:45:51.848902+0000","last_peered":"2024-03-26T21:45:51.848902+0000","last_clean":"2024-03-26T21:45:51.848902+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:45:51.848902+0000","last_undegraded":"2024-03-26T21:45:51.848902+0000","last_fullsized":"2024-03-26T21:45:51.848902+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:43:49.506657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:49.506657+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":751,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.840683+0000","last_change":"2024-03-26T21:44:49.495116+0000","last_active":"2024-03-26T21:45:51.840683+0000","last_peered":"2024-03-26T21:45:51.840683+0000","last_clean":"2024-03-26T21:45:51.840683+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:45:51.840683+0000","last_undegraded":"2024-03-26T21:45:51.840683+0000","last_fullsized":"2024-03-26T21:45:51.840683+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:40:23.862904+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.862904+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":780,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:52.414871+0000","last_change":"2024-03-26T21:45:52.414925+0000","last_active":"2024-03-26T21:45:52.414871+0000","last_peered":"2024-03-26T21:45:52.414871+0000","last_clean":"2024-03-26T21:45:52.414871+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:45:52.414871+0000","last_undegraded":"2024-03-26T21:45:52.414871+0000","last_fullsized":"2024-03-26T21:45:52.414871+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:45:52.414781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:45:52.414781+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:01.246536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"384'5798","reported_seq":9311,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.955897+0000","last_change":"2024-03-26T21:45:54.733920+0000","last_active":"2024-03-26T21:45:56.955897+0000","last_peered":"2024-03-26T21:45:56.955897+0000","last_clean":"2024-03-26T21:45:56.955897+0000","last_became_active":"2024-03-26T21:44:33.349078+0000","last_became_peered":"2024-03-26T21:44:33.349078+0000","last_unstale":"2024-03-26T21:45:56.955897+0000","last_undegraded":"2024-03-26T21:45:56.955897+0000","last_fullsized":"2024-03-26T21:45:56.955897+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'5754","last_scrub_stamp":"2024-03-26T21:45:54.733566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:45:54.733566+0000","objects_scrubbed":2289,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:24.068728+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146735104,"num_objects":2239,"num_object_clones":0,"num_object_copies":6717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841182+0000","last_change":"2024-03-26T21:43:52.448476+0000","last_active":"2024-03-26T21:45:51.841182+0000","last_peered":"2024-03-26T21:45:51.841182+0000","last_clean":"2024-03-26T21:45:51.841182+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:45:51.841182+0000","last_undegraded":"2024-03-26T21:45:51.841182+0000","last_fullsized":"2024-03-26T21:45:51.841182+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:52.447571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:52.447571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":694,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843347+0000","last_change":"2024-03-26T21:42:59.198607+0000","last_active":"2024-03-26T21:45:51.843347+0000","last_peered":"2024-03-26T21:45:51.843347+0000","last_clean":"2024-03-26T21:45:51.843347+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:45:51.843347+0000","last_undegraded":"2024-03-26T21:45:51.843347+0000","last_fullsized":"2024-03-26T21:45:51.843347+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:59.198481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:42:59.198481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845066+0000","last_change":"2024-03-26T21:44:49.426244+0000","last_active":"2024-03-26T21:45:51.845066+0000","last_peered":"2024-03-26T21:45:51.845066+0000","last_clean":"2024-03-26T21:45:51.845066+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:45:51.845066+0000","last_undegraded":"2024-03-26T21:45:51.845066+0000","last_fullsized":"2024-03-26T21:45:51.845066+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:05.251423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:44:05.251423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":749,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848638+0000","last_change":"2024-03-26T21:42:49.300575+0000","last_active":"2024-03-26T21:45:51.848638+0000","last_peered":"2024-03-26T21:45:51.848638+0000","last_clean":"2024-03-26T21:45:51.848638+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:45:51.848638+0000","last_undegraded":"2024-03-26T21:45:51.848638+0000","last_fullsized":"2024-03-26T21:45:51.848638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:42:49.300477+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.300477+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":627,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.851855+0000","last_change":"2024-03-26T21:43:51.167946+0000","last_active":"2024-03-26T21:45:51.851855+0000","last_peered":"2024-03-26T21:45:51.851855+0000","last_clean":"2024-03-26T21:45:51.851855+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:45:51.851855+0000","last_undegraded":"2024-03-26T21:45:51.851855+0000","last_fullsized":"2024-03-26T21:45:51.851855+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:51.167822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:51.167822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":730,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848163+0000","last_change":"2024-03-26T21:43:06.168376+0000","last_active":"2024-03-26T21:45:51.848163+0000","last_peered":"2024-03-26T21:45:51.848163+0000","last_clean":"2024-03-26T21:45:51.848163+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:45:51.848163+0000","last_undegraded":"2024-03-26T21:45:51.848163+0000","last_fullsized":"2024-03-26T21:45:51.848163+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.168150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:43.135432+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.168150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":712,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:54.461685+0000","last_change":"2024-03-26T21:45:54.461723+0000","last_active":"2024-03-26T21:45:54.461685+0000","last_peered":"2024-03-26T21:45:54.461685+0000","last_clean":"2024-03-26T21:45:54.461685+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:45:54.461685+0000","last_undegraded":"2024-03-26T21:45:54.461685+0000","last_fullsized":"2024-03-26T21:45:54.461685+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:54.461604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:45:54.461604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:03.183914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":57,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845911+0000","last_change":"2024-03-26T21:44:49.486181+0000","last_active":"2024-03-26T21:45:51.845911+0000","last_peered":"2024-03-26T21:45:51.845911+0000","last_clean":"2024-03-26T21:45:51.845911+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:45:51.845911+0000","last_undegraded":"2024-03-26T21:45:51.845911+0000","last_fullsized":"2024-03-26T21:45:51.845911+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:40:48.743775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:48.743775+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845611+0000","last_change":"2024-03-26T21:44:49.419484+0000","last_active":"2024-03-26T21:45:51.845611+0000","last_peered":"2024-03-26T21:45:51.845611+0000","last_clean":"2024-03-26T21:45:51.845611+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:45:51.845611+0000","last_undegraded":"2024-03-26T21:45:51.845611+0000","last_fullsized":"2024-03-26T21:45:51.845611+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:39:36.268946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:39:36.268946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":619,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.853548+0000","last_change":"2024-03-26T21:42:58.360038+0000","last_active":"2024-03-26T21:45:51.853548+0000","last_peered":"2024-03-26T21:45:51.853548+0000","last_clean":"2024-03-26T21:45:51.853548+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:45:51.853548+0000","last_undegraded":"2024-03-26T21:45:51.853548+0000","last_fullsized":"2024-03-26T21:45:51.853548+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.359906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.359906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.849795+0000","last_change":"2024-03-26T21:42:50.211297+0000","last_active":"2024-03-26T21:45:51.849795+0000","last_peered":"2024-03-26T21:45:51.849795+0000","last_clean":"2024-03-26T21:45:51.849795+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:45:51.849795+0000","last_undegraded":"2024-03-26T21:45:51.849795+0000","last_fullsized":"2024-03-26T21:45:51.849795+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:50.211214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:42:50.211214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845627+0000","last_change":"2024-03-26T21:44:49.470622+0000","last_active":"2024-03-26T21:45:51.845627+0000","last_peered":"2024-03-26T21:45:51.845627+0000","last_clean":"2024-03-26T21:45:51.845627+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:45:51.845627+0000","last_undegraded":"2024-03-26T21:45:51.845627+0000","last_fullsized":"2024-03-26T21:45:51.845627+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:39:29.152531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:39:29.152531+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":680,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841010+0000","last_change":"2024-03-26T21:40:17.751277+0000","last_active":"2024-03-26T21:45:51.841010+0000","last_peered":"2024-03-26T21:45:51.841010+0000","last_clean":"2024-03-26T21:45:51.841010+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:45:51.841010+0000","last_undegraded":"2024-03-26T21:45:51.841010+0000","last_fullsized":"2024-03-26T21:45:51.841010+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:17.748978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:17.748978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845500+0000","last_change":"2024-03-26T21:44:49.434922+0000","last_active":"2024-03-26T21:45:51.845500+0000","last_peered":"2024-03-26T21:45:51.845500+0000","last_clean":"2024-03-26T21:45:51.845500+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:45:51.845500+0000","last_undegraded":"2024-03-26T21:45:51.845500+0000","last_fullsized":"2024-03-26T21:45:51.845500+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.150652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.150652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":642,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852937+0000","last_change":"2024-03-26T21:44:49.395719+0000","last_active":"2024-03-26T21:45:51.852937+0000","last_peered":"2024-03-26T21:45:51.852937+0000","last_clean":"2024-03-26T21:45:51.852937+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:45:51.852937+0000","last_undegraded":"2024-03-26T21:45:51.852937+0000","last_fullsized":"2024-03-26T21:45:51.852937+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.280112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.280112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":733,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841409+0000","last_change":"2024-03-26T21:43:49.275532+0000","last_active":"2024-03-26T21:45:51.841409+0000","last_peered":"2024-03-26T21:45:51.841409+0000","last_clean":"2024-03-26T21:45:51.841409+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:45:51.841409+0000","last_undegraded":"2024-03-26T21:45:51.841409+0000","last_fullsized":"2024-03-26T21:45:51.841409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:43:49.275284+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:43:49.275284+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845376+0000","last_change":"2024-03-26T21:44:49.421684+0000","last_active":"2024-03-26T21:45:51.845376+0000","last_peered":"2024-03-26T21:45:51.845376+0000","last_clean":"2024-03-26T21:45:51.845376+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:45:51.845376+0000","last_undegraded":"2024-03-26T21:45:51.845376+0000","last_fullsized":"2024-03-26T21:45:51.845376+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:40:49.810420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":724,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841201+0000","last_change":"2024-03-26T21:44:49.398456+0000","last_active":"2024-03-26T21:45:51.841201+0000","last_peered":"2024-03-26T21:45:51.841201+0000","last_clean":"2024-03-26T21:45:51.841201+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:45:51.841201+0000","last_undegraded":"2024-03-26T21:45:51.841201+0000","last_fullsized":"2024-03-26T21:45:51.841201+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:36.141522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:36.141522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":401,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131865+0000","last_change":"2024-03-26T21:41:31.911664+0000","last_active":"2024-03-26T21:46:00.131865+0000","last_peered":"2024-03-26T21:46:00.131865+0000","last_clean":"2024-03-26T21:46:00.131865+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:46:00.131865+0000","last_undegraded":"2024-03-26T21:46:00.131865+0000","last_fullsized":"2024-03-26T21:46:00.131865+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:31.911237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:31.911237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"234'19","reported_seq":424,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131701+0000","last_change":"2024-03-26T21:41:44.918112+0000","last_active":"2024-03-26T21:46:00.131701+0000","last_peered":"2024-03-26T21:46:00.131701+0000","last_clean":"2024-03-26T21:46:00.131701+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:46:00.131701+0000","last_undegraded":"2024-03-26T21:46:00.131701+0000","last_fullsized":"2024-03-26T21:46:00.131701+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:41:44.917970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:41:44.917970+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":708,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841855+0000","last_change":"2024-03-26T21:44:49.485597+0000","last_active":"2024-03-26T21:45:51.841855+0000","last_peered":"2024-03-26T21:45:51.841855+0000","last_clean":"2024-03-26T21:45:51.841855+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:45:51.841855+0000","last_undegraded":"2024-03-26T21:45:51.841855+0000","last_fullsized":"2024-03-26T21:45:51.841855+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:43:48.185306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:48.185306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":909,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850587+0000","last_change":"2024-03-26T21:43:08.196513+0000","last_active":"2024-03-26T21:45:51.850587+0000","last_peered":"2024-03-26T21:45:51.850587+0000","last_clean":"2024-03-26T21:45:51.850587+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:45:51.850587+0000","last_undegraded":"2024-03-26T21:45:51.850587+0000","last_fullsized":"2024-03-26T21:45:51.850587+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:43:08.196428+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.196428+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":637,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852439+0000","last_change":"2024-03-26T21:43:45.070796+0000","last_active":"2024-03-26T21:45:51.852439+0000","last_peered":"2024-03-26T21:45:51.852439+0000","last_clean":"2024-03-26T21:45:51.852439+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:45:51.852439+0000","last_undegraded":"2024-03-26T21:45:51.852439+0000","last_fullsized":"2024-03-26T21:45:51.852439+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:45.070590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:45.070590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":63,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846311+0000","last_change":"2024-03-26T21:44:49.502374+0000","last_active":"2024-03-26T21:45:51.846311+0000","last_peered":"2024-03-26T21:45:51.846311+0000","last_clean":"2024-03-26T21:45:51.846311+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:45:51.846311+0000","last_undegraded":"2024-03-26T21:45:51.846311+0000","last_fullsized":"2024-03-26T21:45:51.846311+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:42:58.972320+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.972320+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":399,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.168057+0000","last_change":"2024-03-26T21:42:54.787575+0000","last_active":"2024-03-26T21:46:00.168057+0000","last_peered":"2024-03-26T21:46:00.168057+0000","last_clean":"2024-03-26T21:46:00.168057+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:46:00.168057+0000","last_undegraded":"2024-03-26T21:46:00.168057+0000","last_fullsized":"2024-03-26T21:46:00.168057+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:54.787464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.787464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"364'13","reported_seq":69,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846744+0000","last_change":"2024-03-26T21:44:49.448360+0000","last_active":"2024-03-26T21:45:51.846744+0000","last_peered":"2024-03-26T21:45:51.846744+0000","last_clean":"2024-03-26T21:45:51.846744+0000","last_became_active":"2024-03-26T21:44:49.402177+0000","last_became_peered":"2024-03-26T21:44:49.402177+0000","last_unstale":"2024-03-26T21:45:51.846744+0000","last_undegraded":"2024-03-26T21:45:51.846744+0000","last_fullsized":"2024-03-26T21:45:51.846744+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-03-26T21:44:06.340020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.340020+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":711,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842060+0000","last_change":"2024-03-26T21:44:49.440707+0000","last_active":"2024-03-26T21:45:51.842060+0000","last_peered":"2024-03-26T21:45:51.842060+0000","last_clean":"2024-03-26T21:45:51.842060+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:45:51.842060+0000","last_undegraded":"2024-03-26T21:45:51.842060+0000","last_fullsized":"2024-03-26T21:45:51.842060+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:07.208520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:07.208520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":763,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850834+0000","last_change":"2024-03-26T21:44:49.444928+0000","last_active":"2024-03-26T21:45:51.850834+0000","last_peered":"2024-03-26T21:45:51.850834+0000","last_clean":"2024-03-26T21:45:51.850834+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:45:51.850834+0000","last_undegraded":"2024-03-26T21:45:51.850834+0000","last_fullsized":"2024-03-26T21:45:51.850834+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:49.162638+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:42:49.162638+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"384'5941","reported_seq":12259,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:45:57.444791+0000","last_change":"2024-03-26T21:45:50.822886+0000","last_active":"2024-03-26T21:45:57.444791+0000","last_peered":"2024-03-26T21:45:57.444791+0000","last_clean":"2024-03-26T21:44:48.321537+0000","last_became_active":"2024-03-26T21:44:50.745053+0000","last_became_peered":"2024-03-26T21:44:50.745053+0000","last_unstale":"2024-03-26T21:45:57.444791+0000","last_undegraded":"2024-03-26T21:44:50.403773+0000","last_fullsized":"2024-03-26T21:44:50.323449+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":854,"num_objects_misplaced":1435,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4293,"num_write_kb":213184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2248,"num_bytes_recovered":147324928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2","objects":80},{"shards":"1,2,6","objects":1435}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":726,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846743+0000","last_change":"2024-03-26T21:43:59.424693+0000","last_active":"2024-03-26T21:45:51.846743+0000","last_peered":"2024-03-26T21:45:51.846743+0000","last_clean":"2024-03-26T21:45:51.846743+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:45:51.846743+0000","last_undegraded":"2024-03-26T21:45:51.846743+0000","last_fullsized":"2024-03-26T21:45:51.846743+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:59.424560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:59.424560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846966+0000","last_change":"2024-03-26T21:44:49.430867+0000","last_active":"2024-03-26T21:45:51.846966+0000","last_peered":"2024-03-26T21:45:51.846966+0000","last_clean":"2024-03-26T21:45:51.846966+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:45:51.846966+0000","last_undegraded":"2024-03-26T21:45:51.846966+0000","last_fullsized":"2024-03-26T21:45:51.846966+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.932318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.932318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":672,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852068+0000","last_change":"2024-03-26T21:45:49.547620+0000","last_active":"2024-03-26T21:45:51.852068+0000","last_peered":"2024-03-26T21:45:51.852068+0000","last_clean":"2024-03-26T21:45:51.852068+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:45:51.852068+0000","last_undegraded":"2024-03-26T21:45:51.852068+0000","last_fullsized":"2024-03-26T21:45:51.852068+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:49.547490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:45:49.547490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:14.948262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":756,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843186+0000","last_change":"2024-03-26T21:44:49.501038+0000","last_active":"2024-03-26T21:45:51.843186+0000","last_peered":"2024-03-26T21:45:51.843186+0000","last_clean":"2024-03-26T21:45:51.843186+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:45:51.843186+0000","last_undegraded":"2024-03-26T21:45:51.843186+0000","last_fullsized":"2024-03-26T21:45:51.843186+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:51.070126+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.070126+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.19","version":"384'5902","reported_seq":14691,"reported_epoch":384,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:45:57.450726+0000","last_change":"2024-03-26T21:45:19.245538+0000","last_active":"2024-03-26T21:45:57.450726+0000","last_peered":"2024-03-26T21:45:57.450726+0000","last_clean":"2024-03-26T21:44:06.345901+0000","last_became_active":"2024-03-26T21:44:50.738290+0000","last_became_peered":"2024-03-26T21:44:50.738290+0000","last_unstale":"2024-03-26T21:45:57.450726+0000","last_undegraded":"2024-03-26T21:44:50.515544+0000","last_fullsized":"2024-03-26T21:44:50.327500+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149520384,"num_objects":2282,"num_object_clones":0,"num_object_copies":6846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":258,"num_objects_misplaced":451,"num_objects_unfound":0,"num_objects_dirty":2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":208544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2633,"num_bytes_recovered":172556288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[2,4],"avail_no_missing":["2","1","4","5","7"],"object_location_counts":[{"shards":"0,1,2,4","objects":193},{"shards":"2,4","objects":1831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":627,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852882+0000","last_change":"2024-03-26T21:43:54.269791+0000","last_active":"2024-03-26T21:45:51.852882+0000","last_peered":"2024-03-26T21:45:51.852882+0000","last_clean":"2024-03-26T21:45:51.852882+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:45:51.852882+0000","last_undegraded":"2024-03-26T21:45:51.852882+0000","last_fullsized":"2024-03-26T21:45:51.852882+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:54.269691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:43:54.269691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":711,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:53.363596+0000","last_change":"2024-03-26T21:45:53.363630+0000","last_active":"2024-03-26T21:45:53.363596+0000","last_peered":"2024-03-26T21:45:53.363596+0000","last_clean":"2024-03-26T21:45:53.363596+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:45:53.363596+0000","last_undegraded":"2024-03-26T21:45:53.363596+0000","last_fullsized":"2024-03-26T21:45:53.363596+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:53.363537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:45:53.363537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:06.000520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847740+0000","last_change":"2024-03-26T21:43:45.525304+0000","last_active":"2024-03-26T21:45:51.847740+0000","last_peered":"2024-03-26T21:45:51.847740+0000","last_clean":"2024-03-26T21:45:51.847740+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:45:51.847740+0000","last_undegraded":"2024-03-26T21:45:51.847740+0000","last_fullsized":"2024-03-26T21:45:51.847740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:45.525163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:45.525163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":59,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846811+0000","last_change":"2024-03-26T21:44:49.468016+0000","last_active":"2024-03-26T21:45:51.846811+0000","last_peered":"2024-03-26T21:45:51.846811+0000","last_clean":"2024-03-26T21:45:51.846811+0000","last_became_active":"2024-03-26T21:44:49.408457+0000","last_became_peered":"2024-03-26T21:44:49.408457+0000","last_unstale":"2024-03-26T21:45:51.846811+0000","last_undegraded":"2024-03-26T21:45:51.846811+0000","last_fullsized":"2024-03-26T21:45:51.846811+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:42:51.267662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:51.267662+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"384'5831","reported_seq":5985,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.927690+0000","last_change":"2024-03-26T21:44:06.625642+0000","last_active":"2024-03-26T21:45:56.927690+0000","last_peered":"2024-03-26T21:45:56.927690+0000","last_clean":"2024-03-26T21:45:56.927690+0000","last_became_active":"2024-03-26T21:44:05.351859+0000","last_became_peered":"2024-03-26T21:44:05.351859+0000","last_unstale":"2024-03-26T21:45:56.927690+0000","last_undegraded":"2024-03-26T21:45:56.927690+0000","last_fullsized":"2024-03-26T21:45:56.927690+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'3258","last_scrub_stamp":"2024-03-26T21:44:06.625360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.625360+0000","objects_scrubbed":1688,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151584768,"num_objects":2313,"num_object_clones":0,"num_object_copies":6939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":102336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":687,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850552+0000","last_change":"2024-03-26T21:43:53.489470+0000","last_active":"2024-03-26T21:45:51.850552+0000","last_peered":"2024-03-26T21:45:51.850552+0000","last_clean":"2024-03-26T21:45:51.850552+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:45:51.850552+0000","last_undegraded":"2024-03-26T21:45:51.850552+0000","last_fullsized":"2024-03-26T21:45:51.850552+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:53.489390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:53.489390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":692,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.853321+0000","last_change":"2024-03-26T21:44:00.293943+0000","last_active":"2024-03-26T21:45:51.853321+0000","last_peered":"2024-03-26T21:45:51.853321+0000","last_clean":"2024-03-26T21:45:51.853321+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:45:51.853321+0000","last_undegraded":"2024-03-26T21:45:51.853321+0000","last_fullsized":"2024-03-26T21:45:51.853321+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:44:00.293821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:44:00.293821+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":747,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.849477+0000","last_change":"2024-03-26T21:42:52.208110+0000","last_active":"2024-03-26T21:45:51.849477+0000","last_peered":"2024-03-26T21:45:51.849477+0000","last_clean":"2024-03-26T21:45:51.849477+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:45:51.849477+0000","last_undegraded":"2024-03-26T21:45:51.849477+0000","last_fullsized":"2024-03-26T21:45:51.849477+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:42:52.207964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.207964+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":855,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:54.520851+0000","last_change":"2024-03-26T21:45:54.520851+0000","last_active":"2024-03-26T21:45:54.520851+0000","last_peered":"2024-03-26T21:45:54.520851+0000","last_clean":"2024-03-26T21:45:54.520851+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:45:54.520851+0000","last_undegraded":"2024-03-26T21:45:54.520851+0000","last_fullsized":"2024-03-26T21:45:54.520851+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:45:54.520765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:45:54.520765+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:01.955680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"384'5800","reported_seq":11671,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.798146+0000","last_change":"2024-03-26T21:45:20.416495+0000","last_active":"2024-03-26T21:45:56.798146+0000","last_peered":"2024-03-26T21:45:56.798146+0000","last_clean":"2024-03-26T21:45:56.798146+0000","last_became_active":"2024-03-26T21:45:20.410428+0000","last_became_peered":"2024-03-26T21:45:20.410428+0000","last_unstale":"2024-03-26T21:45:56.798146+0000","last_undegraded":"2024-03-26T21:45:56.798146+0000","last_fullsized":"2024-03-26T21:45:56.798146+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'3275","last_scrub_stamp":"2024-03-26T21:44:06.904412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.904412+0000","objects_scrubbed":1698,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150274048,"num_objects":2293,"num_object_clones":0,"num_object_copies":6879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":99840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2851,"num_bytes_recovered":186843136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845708+0000","last_change":"2024-03-26T21:44:49.425276+0000","last_active":"2024-03-26T21:45:51.845708+0000","last_peered":"2024-03-26T21:45:51.845708+0000","last_clean":"2024-03-26T21:45:51.845708+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:45:51.845708+0000","last_undegraded":"2024-03-26T21:45:51.845708+0000","last_fullsized":"2024-03-26T21:45:51.845708+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:56.446156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:42:56.446156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":683,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.853212+0000","last_change":"2024-03-26T21:45:38.621541+0000","last_active":"2024-03-26T21:45:51.853212+0000","last_peered":"2024-03-26T21:45:51.853212+0000","last_clean":"2024-03-26T21:45:51.853212+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:45:51.853212+0000","last_undegraded":"2024-03-26T21:45:51.853212+0000","last_fullsized":"2024-03-26T21:45:51.853212+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:45:38.621410+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:45:38.621410+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:59.759096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":463,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131608+0000","last_change":"2024-03-26T21:44:49.491836+0000","last_active":"2024-03-26T21:46:00.131608+0000","last_peered":"2024-03-26T21:46:00.131608+0000","last_clean":"2024-03-26T21:46:00.131608+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:46:00.131608+0000","last_undegraded":"2024-03-26T21:46:00.131608+0000","last_fullsized":"2024-03-26T21:46:00.131608+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:44:00.448307+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_clean_scrub_stamp":"2024-03-26T21:44:00.448307+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845704+0000","last_change":"2024-03-26T21:44:04.372489+0000","last_active":"2024-03-26T21:45:51.845704+0000","last_peered":"2024-03-26T21:45:51.845704+0000","last_clean":"2024-03-26T21:45:51.845704+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:45:51.845704+0000","last_undegraded":"2024-03-26T21:45:51.845704+0000","last_fullsized":"2024-03-26T21:45:51.845704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:04.372337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:44:04.372337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"384'5777","reported_seq":11472,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:58.684111+0000","last_change":"2024-03-26T21:45:51.861673+0000","last_active":"2024-03-26T21:45:58.684111+0000","last_peered":"2024-03-26T21:45:58.684111+0000","last_clean":"2024-03-26T21:45:58.684111+0000","last_became_active":"2024-03-26T21:45:51.847330+0000","last_became_peered":"2024-03-26T21:45:51.847330+0000","last_unstale":"2024-03-26T21:45:58.684111+0000","last_undegraded":"2024-03-26T21:45:58.684111+0000","last_fullsized":"2024-03-26T21:45:58.684111+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'3263","last_scrub_stamp":"2024-03-26T21:44:06.532246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.532246+0000","objects_scrubbed":1632,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146341888,"num_objects":2233,"num_object_clones":0,"num_object_copies":6699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":99712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":167575552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":618,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.851786+0000","last_change":"2024-03-26T21:44:49.396542+0000","last_active":"2024-03-26T21:45:51.851786+0000","last_peered":"2024-03-26T21:45:51.851786+0000","last_clean":"2024-03-26T21:45:51.851786+0000","last_became_active":"2024-03-26T21:44:49.396166+0000","last_became_peered":"2024-03-26T21:44:49.396166+0000","last_unstale":"2024-03-26T21:45:51.851786+0000","last_undegraded":"2024-03-26T21:45:51.851786+0000","last_fullsized":"2024-03-26T21:45:51.851786+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.373726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.373726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847469+0000","last_change":"2024-03-26T21:43:01.366930+0000","last_active":"2024-03-26T21:45:51.847469+0000","last_peered":"2024-03-26T21:45:51.847469+0000","last_clean":"2024-03-26T21:45:51.847469+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:45:51.847469+0000","last_undegraded":"2024-03-26T21:45:51.847469+0000","last_fullsized":"2024-03-26T21:45:51.847469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.366789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":653,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.853059+0000","last_change":"2024-03-26T21:45:46.557694+0000","last_active":"2024-03-26T21:45:51.853059+0000","last_peered":"2024-03-26T21:45:51.853059+0000","last_clean":"2024-03-26T21:45:51.853059+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:45:51.853059+0000","last_undegraded":"2024-03-26T21:45:51.853059+0000","last_fullsized":"2024-03-26T21:45:51.853059+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:46.557567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:45:46.557567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:00.440282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":786,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848283+0000","last_change":"2024-03-26T21:43:55.674688+0000","last_active":"2024-03-26T21:45:51.848283+0000","last_peered":"2024-03-26T21:45:51.848283+0000","last_clean":"2024-03-26T21:45:51.848283+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:45:51.848283+0000","last_undegraded":"2024-03-26T21:45:51.848283+0000","last_fullsized":"2024-03-26T21:45:51.848283+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:43:55.674189+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:43:55.674189+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"384'5734","reported_seq":11384,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.329106+0000","last_change":"2024-03-26T21:45:21.452514+0000","last_active":"2024-03-26T21:45:57.329106+0000","last_peered":"2024-03-26T21:45:57.329106+0000","last_clean":"2024-03-26T21:45:57.329106+0000","last_became_active":"2024-03-26T21:45:21.443678+0000","last_became_peered":"2024-03-26T21:45:21.443678+0000","last_unstale":"2024-03-26T21:45:57.329106+0000","last_undegraded":"2024-03-26T21:45:57.329106+0000","last_fullsized":"2024-03-26T21:45:57.329106+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'3211","last_scrub_stamp":"2024-03-26T21:44:05.713606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:05.713606+0000","objects_scrubbed":1537,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":103360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2743,"num_bytes_recovered":179765248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":689,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842534+0000","last_change":"2024-03-26T21:43:55.321224+0000","last_active":"2024-03-26T21:45:51.842534+0000","last_peered":"2024-03-26T21:45:51.842534+0000","last_clean":"2024-03-26T21:45:51.842534+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:45:51.842534+0000","last_undegraded":"2024-03-26T21:45:51.842534+0000","last_fullsized":"2024-03-26T21:45:51.842534+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:55.321077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:43:55.321077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":733,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.844270+0000","last_change":"2024-03-26T21:44:49.410498+0000","last_active":"2024-03-26T21:45:51.844270+0000","last_peered":"2024-03-26T21:45:51.844270+0000","last_clean":"2024-03-26T21:45:51.844270+0000","last_became_active":"2024-03-26T21:44:49.410172+0000","last_became_peered":"2024-03-26T21:44:49.410172+0000","last_unstale":"2024-03-26T21:45:51.844270+0000","last_undegraded":"2024-03-26T21:45:51.844270+0000","last_fullsized":"2024-03-26T21:45:51.844270+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.277361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":719,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850047+0000","last_change":"2024-03-26T21:44:49.334655+0000","last_active":"2024-03-26T21:45:51.850047+0000","last_peered":"2024-03-26T21:45:51.850047+0000","last_clean":"2024-03-26T21:45:51.850047+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:45:51.850047+0000","last_undegraded":"2024-03-26T21:45:51.850047+0000","last_fullsized":"2024-03-26T21:45:51.850047+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:46.291979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:43:46.291979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":746,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.408991+0000","last_change":"2024-03-26T21:45:56.409040+0000","last_active":"2024-03-26T21:45:56.408991+0000","last_peered":"2024-03-26T21:45:56.408991+0000","last_clean":"2024-03-26T21:45:56.408991+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:45:56.408991+0000","last_undegraded":"2024-03-26T21:45:56.408991+0000","last_fullsized":"2024-03-26T21:45:56.408991+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:45:56.408909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:45:56.408909+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:13.178574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"384'8630","reported_seq":8754,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.738830+0000","last_change":"2024-03-26T21:43:28.830559+0000","last_active":"2024-03-26T21:45:56.738830+0000","last_peered":"2024-03-26T21:45:56.738830+0000","last_clean":"2024-03-26T21:45:56.738830+0000","last_became_active":"2024-03-26T21:43:28.830040+0000","last_became_peered":"2024-03-26T21:43:28.830040+0000","last_unstale":"2024-03-26T21:45:56.738830+0000","last_undegraded":"2024-03-26T21:45:56.738830+0000","last_fullsized":"2024-03-26T21:45:56.738830+0000","mapping_epoch":329,"log_start":"341'3100","ondisk_log_start":"341'3100","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5530,"log_dups_size":0,"ondisk_log_size":5530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304218112,"num_objects":4642,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":424704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846536+0000","last_change":"2024-03-26T21:44:49.402539+0000","last_active":"2024-03-26T21:45:51.846536+0000","last_peered":"2024-03-26T21:45:51.846536+0000","last_clean":"2024-03-26T21:45:51.846536+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:45:51.846536+0000","last_undegraded":"2024-03-26T21:45:51.846536+0000","last_fullsized":"2024-03-26T21:45:51.846536+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.468289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.468289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":718,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850324+0000","last_change":"2024-03-26T21:43:52.436960+0000","last_active":"2024-03-26T21:45:51.850324+0000","last_peered":"2024-03-26T21:45:51.850324+0000","last_clean":"2024-03-26T21:45:51.850324+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:45:51.850324+0000","last_undegraded":"2024-03-26T21:45:51.850324+0000","last_fullsized":"2024-03-26T21:45:51.850324+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:43:52.436837+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:38:57.985228+0000","last_clean_scrub_stamp":"2024-03-26T21:43:52.436837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":136,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847265+0000","last_change":"2024-03-26T21:43:48.307323+0000","last_active":"2024-03-26T21:45:51.847265+0000","last_peered":"2024-03-26T21:45:51.847265+0000","last_clean":"2024-03-26T21:45:51.847265+0000","last_became_active":"2024-03-26T21:43:16.796261+0000","last_became_peered":"2024-03-26T21:43:16.796261+0000","last_unstale":"2024-03-26T21:45:51.847265+0000","last_undegraded":"2024-03-26T21:45:51.847265+0000","last_fullsized":"2024-03-26T21:45:51.847265+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:48.307206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:48.307206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"364'43","reported_seq":982,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.849715+0000","last_change":"2024-03-26T21:44:49.477984+0000","last_active":"2024-03-26T21:45:51.849715+0000","last_peered":"2024-03-26T21:45:51.849715+0000","last_clean":"2024-03-26T21:45:51.849715+0000","last_became_active":"2024-03-26T21:44:49.397229+0000","last_became_peered":"2024-03-26T21:44:49.397229+0000","last_unstale":"2024-03-26T21:45:51.849715+0000","last_undegraded":"2024-03-26T21:45:51.849715+0000","last_fullsized":"2024-03-26T21:45:51.849715+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'42","last_scrub_stamp":"2024-03-26T21:42:55.163584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:55.163584+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842273+0000","last_change":"2024-03-26T21:40:38.776211+0000","last_active":"2024-03-26T21:45:51.842273+0000","last_peered":"2024-03-26T21:45:51.842273+0000","last_clean":"2024-03-26T21:45:51.842273+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:45:51.842273+0000","last_undegraded":"2024-03-26T21:45:51.842273+0000","last_fullsized":"2024-03-26T21:45:51.842273+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:38.774895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:38.774895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"384'6008","reported_seq":6163,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.905699+0000","last_change":"2024-03-26T21:44:05.834140+0000","last_active":"2024-03-26T21:45:56.905699+0000","last_peered":"2024-03-26T21:45:56.905699+0000","last_clean":"2024-03-26T21:45:56.905699+0000","last_became_active":"2024-03-26T21:44:05.344228+0000","last_became_peered":"2024-03-26T21:44:05.344228+0000","last_unstale":"2024-03-26T21:45:56.905699+0000","last_undegraded":"2024-03-26T21:45:56.905699+0000","last_fullsized":"2024-03-26T21:45:56.905699+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'3359","last_scrub_stamp":"2024-03-26T21:44:05.833826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:05.833826+0000","objects_scrubbed":1710,"log_size":6008,"log_dups_size":0,"ondisk_log_size":6008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2658,"num_write_kb":106304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":618,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:55.605517+0000","last_change":"2024-03-26T21:45:55.605564+0000","last_active":"2024-03-26T21:45:55.605517+0000","last_peered":"2024-03-26T21:45:55.605517+0000","last_clean":"2024-03-26T21:45:55.605517+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:45:55.605517+0000","last_undegraded":"2024-03-26T21:45:55.605517+0000","last_fullsized":"2024-03-26T21:45:55.605517+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:55.605391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:36.751794+0000","last_clean_scrub_stamp":"2024-03-26T21:45:55.605391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:57.709749+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":717,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.840762+0000","last_change":"2024-03-26T21:40:46.085237+0000","last_active":"2024-03-26T21:45:51.840762+0000","last_peered":"2024-03-26T21:45:51.840762+0000","last_clean":"2024-03-26T21:45:51.840762+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:45:51.840762+0000","last_undegraded":"2024-03-26T21:45:51.840762+0000","last_fullsized":"2024-03-26T21:45:51.840762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:46.085013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.599655+0000","last_clean_scrub_stamp":"2024-03-26T21:40:46.085013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":736,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843658+0000","last_change":"2024-03-26T21:44:49.410666+0000","last_active":"2024-03-26T21:45:51.843658+0000","last_peered":"2024-03-26T21:45:51.843658+0000","last_clean":"2024-03-26T21:45:51.843658+0000","last_became_active":"2024-03-26T21:44:49.410374+0000","last_became_peered":"2024-03-26T21:44:49.410374+0000","last_unstale":"2024-03-26T21:45:51.843658+0000","last_undegraded":"2024-03-26T21:45:51.843658+0000","last_fullsized":"2024-03-26T21:45:51.843658+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:50.270808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:43:50.270808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":751,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841698+0000","last_change":"2024-03-26T21:43:57.459826+0000","last_active":"2024-03-26T21:45:51.841698+0000","last_peered":"2024-03-26T21:45:51.841698+0000","last_clean":"2024-03-26T21:45:51.841698+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:45:51.841698+0000","last_undegraded":"2024-03-26T21:45:51.841698+0000","last_fullsized":"2024-03-26T21:45:51.841698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:43:57.459656+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:43:57.459656+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":710,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846633+0000","last_change":"2024-03-26T21:43:43.406078+0000","last_active":"2024-03-26T21:45:51.846633+0000","last_peered":"2024-03-26T21:45:51.846633+0000","last_clean":"2024-03-26T21:45:51.846633+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:45:51.846633+0000","last_undegraded":"2024-03-26T21:45:51.846633+0000","last_fullsized":"2024-03-26T21:45:51.846633+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:43.405952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:43:43.405952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"384'8462","reported_seq":8586,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.755814+0000","last_change":"2024-03-26T21:43:28.828609+0000","last_active":"2024-03-26T21:45:56.755814+0000","last_peered":"2024-03-26T21:45:56.755814+0000","last_clean":"2024-03-26T21:45:56.755814+0000","last_became_active":"2024-03-26T21:43:28.827990+0000","last_became_peered":"2024-03-26T21:43:28.827990+0000","last_unstale":"2024-03-26T21:45:56.755814+0000","last_undegraded":"2024-03-26T21:45:56.755814+0000","last_fullsized":"2024-03-26T21:45:56.755814+0000","mapping_epoch":329,"log_start":"341'3000","ondisk_log_start":"341'3000","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296615936,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":415616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850883+0000","last_change":"2024-03-26T21:45:50.776522+0000","last_active":"2024-03-26T21:45:51.850883+0000","last_peered":"2024-03-26T21:45:51.850883+0000","last_clean":"2024-03-26T21:45:51.850883+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:45:51.850883+0000","last_undegraded":"2024-03-26T21:45:51.850883+0000","last_fullsized":"2024-03-26T21:45:51.850883+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:50.776372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:45:50.776372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:04.176194+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":898,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.853568+0000","last_change":"2024-03-26T21:45:36.556316+0000","last_active":"2024-03-26T21:45:51.853568+0000","last_peered":"2024-03-26T21:45:51.853568+0000","last_clean":"2024-03-26T21:45:51.853568+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:45:51.853568+0000","last_undegraded":"2024-03-26T21:45:51.853568+0000","last_fullsized":"2024-03-26T21:45:51.853568+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:45:36.556160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:45:36.556160+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:46:55.583965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845451+0000","last_change":"2024-03-26T21:44:49.400558+0000","last_active":"2024-03-26T21:45:51.845451+0000","last_peered":"2024-03-26T21:45:51.845451+0000","last_clean":"2024-03-26T21:45:51.845451+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:45:51.845451+0000","last_undegraded":"2024-03-26T21:45:51.845451+0000","last_fullsized":"2024-03-26T21:45:51.845451+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:40:41.689096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":1033,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848195+0000","last_change":"2024-03-26T21:43:03.102559+0000","last_active":"2024-03-26T21:45:51.848195+0000","last_peered":"2024-03-26T21:45:51.848195+0000","last_clean":"2024-03-26T21:45:51.848195+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:45:51.848195+0000","last_undegraded":"2024-03-26T21:45:51.848195+0000","last_fullsized":"2024-03-26T21:45:51.848195+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:43:03.102463+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:43:03.102463+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":638,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852742+0000","last_change":"2024-03-26T21:45:37.589750+0000","last_active":"2024-03-26T21:45:51.852742+0000","last_peered":"2024-03-26T21:45:51.852742+0000","last_clean":"2024-03-26T21:45:51.852742+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:45:51.852742+0000","last_undegraded":"2024-03-26T21:45:51.852742+0000","last_fullsized":"2024-03-26T21:45:51.852742+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:45:37.589632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:45:37.589632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:47:05.967504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"384'5934","reported_seq":6142,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:58.894343+0000","last_change":"2024-03-26T21:44:09.418482+0000","last_active":"2024-03-26T21:45:58.894343+0000","last_peered":"2024-03-26T21:45:58.894343+0000","last_clean":"2024-03-26T21:45:58.894343+0000","last_became_active":"2024-03-26T21:44:05.355631+0000","last_became_peered":"2024-03-26T21:44:05.355631+0000","last_unstale":"2024-03-26T21:45:58.894343+0000","last_undegraded":"2024-03-26T21:45:58.894343+0000","last_fullsized":"2024-03-26T21:45:58.894343+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'3377","last_scrub_stamp":"2024-03-26T21:44:09.418344+0000","last_deep_scrub":"349'3377","last_deep_scrub_stamp":"2024-03-26T21:44:09.418344+0000","last_clean_scrub_stamp":"2024-03-26T21:44:09.418344+0000","objects_scrubbed":1725,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152174592,"num_objects":2322,"num_object_clones":0,"num_object_copies":6966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2583,"num_write_kb":101760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":717,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.840498+0000","last_change":"2024-03-26T21:44:49.442641+0000","last_active":"2024-03-26T21:45:51.840498+0000","last_peered":"2024-03-26T21:45:51.840498+0000","last_clean":"2024-03-26T21:45:51.840498+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:45:51.840498+0000","last_undegraded":"2024-03-26T21:45:51.840498+0000","last_fullsized":"2024-03-26T21:45:51.840498+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:58.461867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:58.461867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":726,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842323+0000","last_change":"2024-03-26T21:44:49.439280+0000","last_active":"2024-03-26T21:45:51.842323+0000","last_peered":"2024-03-26T21:45:51.842323+0000","last_clean":"2024-03-26T21:45:51.842323+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:45:51.842323+0000","last_undegraded":"2024-03-26T21:45:51.842323+0000","last_fullsized":"2024-03-26T21:45:51.842323+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:40:47.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1591,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.235390+0000","last_change":"2024-03-26T21:44:07.308336+0000","last_active":"2024-03-26T21:45:57.235390+0000","last_peered":"2024-03-26T21:45:57.235390+0000","last_clean":"2024-03-26T21:45:57.235390+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:45:57.235390+0000","last_undegraded":"2024-03-26T21:45:57.235390+0000","last_fullsized":"2024-03-26T21:45:57.235390+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:44:07.308250+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:44:07.308250+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":923,"num_read_kb":918,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":745,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843017+0000","last_change":"2024-03-26T21:40:16.682329+0000","last_active":"2024-03-26T21:45:51.843017+0000","last_peered":"2024-03-26T21:45:51.843017+0000","last_clean":"2024-03-26T21:45:51.843017+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:45:51.843017+0000","last_undegraded":"2024-03-26T21:45:51.843017+0000","last_fullsized":"2024-03-26T21:45:51.843017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:40:16.682154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:16.682154+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":700,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.851116+0000","last_change":"2024-03-26T21:44:07.472290+0000","last_active":"2024-03-26T21:45:51.851116+0000","last_peered":"2024-03-26T21:45:51.851116+0000","last_clean":"2024-03-26T21:45:51.851116+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:45:51.851116+0000","last_undegraded":"2024-03-26T21:45:51.851116+0000","last_fullsized":"2024-03-26T21:45:51.851116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:07.472213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:44:07.472213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"384'8520","reported_seq":8645,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.784601+0000","last_change":"2024-03-26T21:43:28.830803+0000","last_active":"2024-03-26T21:45:56.784601+0000","last_peered":"2024-03-26T21:45:56.784601+0000","last_clean":"2024-03-26T21:45:56.784601+0000","last_became_active":"2024-03-26T21:43:28.829451+0000","last_became_peered":"2024-03-26T21:43:28.829451+0000","last_unstale":"2024-03-26T21:45:56.784601+0000","last_undegraded":"2024-03-26T21:45:56.784601+0000","last_fullsized":"2024-03-26T21:45:56.784601+0000","mapping_epoch":329,"log_start":"342'3000","ondisk_log_start":"342'3000","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300613656,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8520,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":697,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.840584+0000","last_change":"2024-03-26T21:43:51.351250+0000","last_active":"2024-03-26T21:45:51.840584+0000","last_peered":"2024-03-26T21:45:51.840584+0000","last_clean":"2024-03-26T21:45:51.840584+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:45:51.840584+0000","last_undegraded":"2024-03-26T21:45:51.840584+0000","last_fullsized":"2024-03-26T21:45:51.840584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:51.351088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:51.351088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":717,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842182+0000","last_change":"2024-03-26T21:43:56.440663+0000","last_active":"2024-03-26T21:45:51.842182+0000","last_peered":"2024-03-26T21:45:51.842182+0000","last_clean":"2024-03-26T21:45:51.842182+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:45:51.842182+0000","last_undegraded":"2024-03-26T21:45:51.842182+0000","last_fullsized":"2024-03-26T21:45:51.842182+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.440543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.440543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":749,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843503+0000","last_change":"2024-03-26T21:43:04.127568+0000","last_active":"2024-03-26T21:45:51.843503+0000","last_peered":"2024-03-26T21:45:51.843503+0000","last_clean":"2024-03-26T21:45:51.843503+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:45:51.843503+0000","last_undegraded":"2024-03-26T21:45:51.843503+0000","last_fullsized":"2024-03-26T21:45:51.843503+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:43:04.127429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:43:04.127429+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":709,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.848338+0000","last_change":"2024-03-26T21:43:58.611970+0000","last_active":"2024-03-26T21:45:51.848338+0000","last_peered":"2024-03-26T21:45:51.848338+0000","last_clean":"2024-03-26T21:45:51.848338+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:45:51.848338+0000","last_undegraded":"2024-03-26T21:45:51.848338+0000","last_fullsized":"2024-03-26T21:45:51.848338+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:58.611763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:43:58.611763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"384'8425","reported_seq":8549,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:58.263240+0000","last_change":"2024-03-26T21:43:28.831132+0000","last_active":"2024-03-26T21:45:58.263240+0000","last_peered":"2024-03-26T21:45:58.263240+0000","last_clean":"2024-03-26T21:45:58.263240+0000","last_became_active":"2024-03-26T21:43:28.830782+0000","last_became_peered":"2024-03-26T21:43:28.830782+0000","last_unstale":"2024-03-26T21:45:58.263240+0000","last_undegraded":"2024-03-26T21:45:58.263240+0000","last_fullsized":"2024-03-26T21:45:58.263240+0000","mapping_epoch":329,"log_start":"349'3500","ondisk_log_start":"349'3500","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299171840,"num_objects":4565,"num_object_clones":0,"num_object_copies":13695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":415680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":693,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841112+0000","last_change":"2024-03-26T21:44:49.437119+0000","last_active":"2024-03-26T21:45:51.841112+0000","last_peered":"2024-03-26T21:45:51.841112+0000","last_clean":"2024-03-26T21:45:51.841112+0000","last_became_active":"2024-03-26T21:44:49.436816+0000","last_became_peered":"2024-03-26T21:44:49.436816+0000","last_unstale":"2024-03-26T21:45:51.841112+0000","last_undegraded":"2024-03-26T21:45:51.841112+0000","last_fullsized":"2024-03-26T21:45:51.841112+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:28.770387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:40:28.770387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850646+0000","last_change":"2024-03-26T21:44:49.403663+0000","last_active":"2024-03-26T21:45:51.850646+0000","last_peered":"2024-03-26T21:45:51.850646+0000","last_clean":"2024-03-26T21:45:51.850646+0000","last_became_active":"2024-03-26T21:44:49.403379+0000","last_became_peered":"2024-03-26T21:44:49.403379+0000","last_unstale":"2024-03-26T21:45:51.850646+0000","last_undegraded":"2024-03-26T21:45:51.850646+0000","last_fullsized":"2024-03-26T21:45:51.850646+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:54.431822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:43:54.431822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":772,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841298+0000","last_change":"2024-03-26T21:40:29.968002+0000","last_active":"2024-03-26T21:45:51.841298+0000","last_peered":"2024-03-26T21:45:51.841298+0000","last_clean":"2024-03-26T21:45:51.841298+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:45:51.841298+0000","last_undegraded":"2024-03-26T21:45:51.841298+0000","last_fullsized":"2024-03-26T21:45:51.841298+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:40:29.967581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:29.967581+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":902,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842018+0000","last_change":"2024-03-26T21:40:27.726654+0000","last_active":"2024-03-26T21:45:51.842018+0000","last_peered":"2024-03-26T21:45:51.842018+0000","last_clean":"2024-03-26T21:45:51.842018+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:45:51.842018+0000","last_undegraded":"2024-03-26T21:45:51.842018+0000","last_fullsized":"2024-03-26T21:45:51.842018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:40:27.726492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:27.726492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"384'8458","reported_seq":8591,"reported_epoch":384,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-26T21:45:56.800816+0000","last_change":"2024-03-26T21:45:50.333960+0000","last_active":"2024-03-26T21:45:56.800816+0000","last_peered":"2024-03-26T21:45:56.800816+0000","last_clean":"2024-03-26T21:44:48.303243+0000","last_became_active":"2024-03-26T21:44:55.642635+0000","last_became_peered":"2024-03-26T21:44:55.642635+0000","last_unstale":"2024-03-26T21:45:56.800816+0000","last_undegraded":"2024-03-26T21:45:56.800816+0000","last_fullsized":"2024-03-26T21:45:56.800816+0000","mapping_epoch":364,"log_start":"342'3000","ondisk_log_start":"342'3000","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":5458,"log_dups_size":0,"ondisk_log_size":5458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302907392,"num_objects":4622,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4622,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8458,"num_write_kb":418560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":4622}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":613,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.852306+0000","last_change":"2024-03-26T21:43:00.377290+0000","last_active":"2024-03-26T21:45:51.852306+0000","last_peered":"2024-03-26T21:45:51.852306+0000","last_clean":"2024-03-26T21:45:51.852306+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:45:51.852306+0000","last_undegraded":"2024-03-26T21:45:51.852306+0000","last_fullsized":"2024-03-26T21:45:51.852306+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:00.377185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:00.377185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":707,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.843787+0000","last_change":"2024-03-26T21:39:45.352141+0000","last_active":"2024-03-26T21:45:51.843787+0000","last_peered":"2024-03-26T21:45:51.843787+0000","last_clean":"2024-03-26T21:45:51.843787+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:45:51.843787+0000","last_undegraded":"2024-03-26T21:45:51.843787+0000","last_fullsized":"2024-03-26T21:45:51.843787+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:39:45.352017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:39:45.352017+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":438,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131554+0000","last_change":"2024-03-26T21:42:52.675227+0000","last_active":"2024-03-26T21:46:00.131554+0000","last_peered":"2024-03-26T21:46:00.131554+0000","last_clean":"2024-03-26T21:46:00.131554+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:46:00.131554+0000","last_undegraded":"2024-03-26T21:46:00.131554+0000","last_fullsized":"2024-03-26T21:46:00.131554+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:42:52.674814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:52.674814+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":69,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846097+0000","last_change":"2024-03-26T21:44:49.488792+0000","last_active":"2024-03-26T21:45:51.846097+0000","last_peered":"2024-03-26T21:45:51.846097+0000","last_clean":"2024-03-26T21:45:51.846097+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:45:51.846097+0000","last_undegraded":"2024-03-26T21:45:51.846097+0000","last_fullsized":"2024-03-26T21:45:51.846097+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:40:23.374040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:23.374040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"386'8438","reported_seq":8566,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:01.001728+0000","last_change":"2024-03-26T21:43:28.833912+0000","last_active":"2024-03-26T21:46:01.001728+0000","last_peered":"2024-03-26T21:46:01.001728+0000","last_clean":"2024-03-26T21:46:01.001728+0000","last_became_active":"2024-03-26T21:43:28.833363+0000","last_became_peered":"2024-03-26T21:43:28.833363+0000","last_unstale":"2024-03-26T21:46:01.001728+0000","last_undegraded":"2024-03-26T21:46:01.001728+0000","last_fullsized":"2024-03-26T21:46:01.001728+0000","mapping_epoch":329,"log_start":"354'4200","ondisk_log_start":"354'4200","created":329,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:43:27.810562+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":408704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":421,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131644+0000","last_change":"2024-03-26T21:44:49.395042+0000","last_active":"2024-03-26T21:46:00.131644+0000","last_peered":"2024-03-26T21:46:00.131644+0000","last_clean":"2024-03-26T21:46:00.131644+0000","last_became_active":"2024-03-26T21:44:49.394613+0000","last_became_peered":"2024-03-26T21:44:49.394613+0000","last_unstale":"2024-03-26T21:46:00.131644+0000","last_undegraded":"2024-03-26T21:46:00.131644+0000","last_fullsized":"2024-03-26T21:46:00.131644+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:04.596247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:44:04.596247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847168+0000","last_change":"2024-03-26T21:44:49.417965+0000","last_active":"2024-03-26T21:45:51.847168+0000","last_peered":"2024-03-26T21:45:51.847168+0000","last_clean":"2024-03-26T21:45:51.847168+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:45:51.847168+0000","last_undegraded":"2024-03-26T21:45:51.847168+0000","last_fullsized":"2024-03-26T21:45:51.847168+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:08.323649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.323649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"317'38","reported_seq":69,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845804+0000","last_change":"2024-03-26T21:44:49.479375+0000","last_active":"2024-03-26T21:45:51.845804+0000","last_peered":"2024-03-26T21:45:51.845804+0000","last_clean":"2024-03-26T21:45:51.845804+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:45:51.845804+0000","last_undegraded":"2024-03-26T21:45:51.845804+0000","last_fullsized":"2024-03-26T21:45:51.845804+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:42:54.348885+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":420,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131721+0000","last_change":"2024-03-26T21:44:49.411632+0000","last_active":"2024-03-26T21:46:00.131721+0000","last_peered":"2024-03-26T21:46:00.131721+0000","last_clean":"2024-03-26T21:46:00.131721+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:46:00.131721+0000","last_undegraded":"2024-03-26T21:46:00.131721+0000","last_fullsized":"2024-03-26T21:46:00.131721+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:57.506416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:43:57.506416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"386'5835","reported_seq":5994,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.754866+0000","last_change":"2024-03-26T21:44:06.129131+0000","last_active":"2024-03-26T21:46:00.754866+0000","last_peered":"2024-03-26T21:46:00.754866+0000","last_clean":"2024-03-26T21:46:00.754866+0000","last_became_active":"2024-03-26T21:44:05.344991+0000","last_became_peered":"2024-03-26T21:44:05.344991+0000","last_unstale":"2024-03-26T21:46:00.754866+0000","last_undegraded":"2024-03-26T21:46:00.754866+0000","last_fullsized":"2024-03-26T21:46:00.754866+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'3302","last_scrub_stamp":"2024-03-26T21:44:06.128807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.128807+0000","objects_scrubbed":1676,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":490,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":6585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":98176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":687,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.850324+0000","last_change":"2024-03-26T21:43:01.304736+0000","last_active":"2024-03-26T21:45:51.850324+0000","last_peered":"2024-03-26T21:45:51.850324+0000","last_clean":"2024-03-26T21:45:51.850324+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:45:51.850324+0000","last_undegraded":"2024-03-26T21:45:51.850324+0000","last_fullsized":"2024-03-26T21:45:51.850324+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:01.304625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:43:01.304625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":524,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.132346+0000","last_change":"2024-03-26T21:43:58.486577+0000","last_active":"2024-03-26T21:46:00.132346+0000","last_peered":"2024-03-26T21:46:00.132346+0000","last_clean":"2024-03-26T21:46:00.132346+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:46:00.132346+0000","last_undegraded":"2024-03-26T21:46:00.132346+0000","last_fullsized":"2024-03-26T21:46:00.132346+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:43:58.486323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:58.486323+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846012+0000","last_change":"2024-03-26T21:44:49.426956+0000","last_active":"2024-03-26T21:45:51.846012+0000","last_peered":"2024-03-26T21:45:51.846012+0000","last_clean":"2024-03-26T21:45:51.846012+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:45:51.846012+0000","last_undegraded":"2024-03-26T21:45:51.846012+0000","last_fullsized":"2024-03-26T21:45:51.846012+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:43:06.961290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":840,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.841503+0000","last_change":"2024-03-26T21:44:49.500635+0000","last_active":"2024-03-26T21:45:51.841503+0000","last_peered":"2024-03-26T21:45:51.841503+0000","last_clean":"2024-03-26T21:45:51.841503+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:45:51.841503+0000","last_undegraded":"2024-03-26T21:45:51.841503+0000","last_fullsized":"2024-03-26T21:45:51.841503+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:40:14.873121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:40:14.873121+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"384'5821","reported_seq":5976,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:56.788666+0000","last_change":"2024-03-26T21:44:06.194802+0000","last_active":"2024-03-26T21:45:56.788666+0000","last_peered":"2024-03-26T21:45:56.788666+0000","last_clean":"2024-03-26T21:45:56.788666+0000","last_became_active":"2024-03-26T21:44:05.345705+0000","last_became_peered":"2024-03-26T21:44:05.345705+0000","last_unstale":"2024-03-26T21:45:56.788666+0000","last_undegraded":"2024-03-26T21:45:56.788666+0000","last_fullsized":"2024-03-26T21:45:56.788666+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'3249","last_scrub_stamp":"2024-03-26T21:44:06.194523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.194523+0000","objects_scrubbed":1658,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150536192,"num_objects":2297,"num_object_clones":0,"num_object_copies":6891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":102976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846416+0000","last_change":"2024-03-26T21:44:49.392525+0000","last_active":"2024-03-26T21:45:51.846416+0000","last_peered":"2024-03-26T21:45:51.846416+0000","last_clean":"2024-03-26T21:45:51.846416+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:45:51.846416+0000","last_undegraded":"2024-03-26T21:45:51.846416+0000","last_fullsized":"2024-03-26T21:45:51.846416+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:59.627397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:43:59.627397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.851073+0000","last_change":"2024-03-26T21:43:05.204186+0000","last_active":"2024-03-26T21:45:51.851073+0000","last_peered":"2024-03-26T21:45:51.851073+0000","last_clean":"2024-03-26T21:45:51.851073+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:45:51.851073+0000","last_undegraded":"2024-03-26T21:45:51.851073+0000","last_fullsized":"2024-03-26T21:45:51.851073+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:05.204079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:43:05.204079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":402,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131499+0000","last_change":"2024-03-26T21:41:33.902737+0000","last_active":"2024-03-26T21:46:00.131499+0000","last_peered":"2024-03-26T21:46:00.131499+0000","last_clean":"2024-03-26T21:46:00.131499+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:46:00.131499+0000","last_undegraded":"2024-03-26T21:46:00.131499+0000","last_fullsized":"2024-03-26T21:46:00.131499+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:33.902465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:41:33.902465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"234'19","reported_seq":425,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.132337+0000","last_change":"2024-03-26T21:42:57.692467+0000","last_active":"2024-03-26T21:46:00.132337+0000","last_peered":"2024-03-26T21:46:00.132337+0000","last_clean":"2024-03-26T21:46:00.132337+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:46:00.132337+0000","last_undegraded":"2024-03-26T21:46:00.132337+0000","last_fullsized":"2024-03-26T21:46:00.132337+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:42:57.692111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:42:57.692111+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"384'5835","reported_seq":11607,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.278805+0000","last_change":"2024-03-26T21:45:51.867897+0000","last_active":"2024-03-26T21:45:57.278805+0000","last_peered":"2024-03-26T21:45:57.278805+0000","last_clean":"2024-03-26T21:45:57.278805+0000","last_became_active":"2024-03-26T21:45:51.852274+0000","last_became_peered":"2024-03-26T21:45:51.852274+0000","last_unstale":"2024-03-26T21:45:57.278805+0000","last_undegraded":"2024-03-26T21:45:57.278805+0000","last_fullsized":"2024-03-26T21:45:57.278805+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'3299","last_scrub_stamp":"2024-03-26T21:44:07.311184+0000","last_deep_scrub":"346'3299","last_deep_scrub_stamp":"2024-03-26T21:44:07.311184+0000","last_clean_scrub_stamp":"2024-03-26T21:44:07.311184+0000","objects_scrubbed":1628,"log_size":5835,"log_dups_size":0,"ondisk_log_size":5835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144834560,"num_objects":2210,"num_object_clones":0,"num_object_copies":6630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":99776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":167313408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":729,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.844183+0000","last_change":"2024-03-26T21:44:49.411355+0000","last_active":"2024-03-26T21:45:51.844183+0000","last_peered":"2024-03-26T21:45:51.844183+0000","last_clean":"2024-03-26T21:45:51.844183+0000","last_became_active":"2024-03-26T21:44:49.410905+0000","last_became_peered":"2024-03-26T21:44:49.410905+0000","last_unstale":"2024-03-26T21:45:51.844183+0000","last_undegraded":"2024-03-26T21:45:51.844183+0000","last_fullsized":"2024-03-26T21:45:51.844183+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:42:58.401600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:42:58.401600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":722,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.842320+0000","last_change":"2024-03-26T21:40:22.891092+0000","last_active":"2024-03-26T21:45:51.842320+0000","last_peered":"2024-03-26T21:45:51.842320+0000","last_clean":"2024-03-26T21:45:51.842320+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:45:51.842320+0000","last_undegraded":"2024-03-26T21:45:51.842320+0000","last_fullsized":"2024-03-26T21:45:51.842320+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:22.890287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:40:22.890287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":738,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.847980+0000","last_change":"2024-03-26T21:43:56.623512+0000","last_active":"2024-03-26T21:45:51.847980+0000","last_peered":"2024-03-26T21:45:51.847980+0000","last_clean":"2024-03-26T21:45:51.847980+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:45:51.847980+0000","last_undegraded":"2024-03-26T21:45:51.847980+0000","last_fullsized":"2024-03-26T21:45:51.847980+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.623203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.623203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":448,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.132408+0000","last_change":"2024-03-26T21:41:50.859131+0000","last_active":"2024-03-26T21:46:00.132408+0000","last_peered":"2024-03-26T21:46:00.132408+0000","last_clean":"2024-03-26T21:46:00.132408+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:46:00.132408+0000","last_undegraded":"2024-03-26T21:46:00.132408+0000","last_fullsized":"2024-03-26T21:46:00.132408+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:41:50.858942+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"384'5909","reported_seq":6063,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:57.752107+0000","last_change":"2024-03-26T21:44:05.773577+0000","last_active":"2024-03-26T21:45:57.752107+0000","last_peered":"2024-03-26T21:45:57.752107+0000","last_clean":"2024-03-26T21:45:57.752107+0000","last_became_active":"2024-03-26T21:44:05.340609+0000","last_became_peered":"2024-03-26T21:44:05.340609+0000","last_unstale":"2024-03-26T21:45:57.752107+0000","last_undegraded":"2024-03-26T21:45:57.752107+0000","last_fullsized":"2024-03-26T21:45:57.752107+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":329,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'3335","last_scrub_stamp":"2024-03-26T21:44:05.773136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:27.810562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:05.773136+0000","objects_scrubbed":1681,"log_size":5909,"log_dups_size":0,"ondisk_log_size":5909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153092096,"num_objects":2336,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":103424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":401,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.132271+0000","last_change":"2024-03-26T21:41:56.818280+0000","last_active":"2024-03-26T21:46:00.132271+0000","last_peered":"2024-03-26T21:46:00.132271+0000","last_clean":"2024-03-26T21:46:00.132271+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:46:00.132271+0000","last_undegraded":"2024-03-26T21:46:00.132271+0000","last_fullsized":"2024-03-26T21:46:00.132271+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:41:56.817883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":929,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.846558+0000","last_change":"2024-03-26T21:42:45.315356+0000","last_active":"2024-03-26T21:45:51.846558+0000","last_peered":"2024-03-26T21:45:51.846558+0000","last_clean":"2024-03-26T21:45:51.846558+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:45:51.846558+0000","last_undegraded":"2024-03-26T21:45:51.846558+0000","last_fullsized":"2024-03-26T21:45:51.846558+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:42:45.315022+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:42:45.315022+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T21:45:51.845801+0000","last_change":"2024-03-26T21:44:49.416624+0000","last_active":"2024-03-26T21:45:51.845801+0000","last_peered":"2024-03-26T21:45:51.845801+0000","last_clean":"2024-03-26T21:45:51.845801+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:45:51.845801+0000","last_undegraded":"2024-03-26T21:45:51.845801+0000","last_fullsized":"2024-03-26T21:45:51.845801+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:40:40.653184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:40:40.653184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":482,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-26T21:46:00.131895+0000","last_change":"2024-03-26T21:44:49.481986+0000","last_active":"2024-03-26T21:46:00.131895+0000","last_peered":"2024-03-26T21:46:00.131895+0000","last_clean":"2024-03-26T21:46:00.131895+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:46:00.131895+0000","last_undegraded":"2024-03-26T21:46:00.131895+0000","last_fullsized":"2024-03-26T21:46:00.131895+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:43:08.600649+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":4767219736,"num_objects":72745,"num_object_clones":0,"num_object_copies":218235,"num_objects_missing_on_primary":656,"num_objects_missing":656,"num_objects_degraded":1530,"num_objects_misplaced":8985,"num_objects_unfound":0,"num_objects_dirty":72745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114637,"num_write_kb":5390529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29885,"num_bytes_recovered":1930100736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15206723584,"data_stored":15206711368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148022,"ondisk_log_size":148022,"up":78,"acting":75,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":923,"num_read_kb":918,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1112,"num_read_kb":869,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973752,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618736,"kb_used_data":1290344,"kb_used_omap":6,"kb_used_meta":328377,"kb_avail":92126416,"statfs":{"total":95995035648,"available":94337449984,"internally_reserved":0,"allocated":1321312256,"data_stored":1320444751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":336258344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365019,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3003284,"kb_used_data":2530572,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":90741868,"statfs":{"total":95995035648,"available":92919672832,"internally_reserved":0,"allocated":2591305728,"data_stored":2590464507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2214012,"kb_used_data":1560884,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91531140,"statfs":{"total":95995035648,"available":93727887360,"internally_reserved":0,"allocated":1598345216,"data_stored":1597464220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738618,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2971960,"kb_used_data":2268848,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":90773192,"statfs":{"total":95995035648,"available":92951748608,"internally_reserved":0,"allocated":2323300352,"data_stored":2322424095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411567,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2712456,"kb_used_data":2099584,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91032696,"statfs":{"total":95995035648,"available":93217480704,"internally_reserved":0,"allocated":2149974016,"data_stored":2149144630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149929,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1676148,"kb_used_data":932780,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92069004,"statfs":{"total":95995035648,"available":94278660096,"internally_reserved":0,"allocated":955166720,"data_stored":954293067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":761199047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650526,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3105684,"kb_used_data":2554308,"kb_used_omap":4,"kb_used_meta":551355,"kb_avail":90639468,"statfs":{"total":95995035648,"available":92814815232,"internally_reserved":0,"allocated":2615611392,"data_stored":2614728268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":564588195},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659358,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2316512,"kb_used_data":1681432,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91428640,"statfs":{"total":95995035648,"available":93622927360,"internally_reserved":0,"allocated":1721786368,"data_stored":1720932975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2314342400,"data_stored":2314338328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1592852480,"data_stored":1592852480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2133000192,"data_stored":2133000192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":960692224,"data_stored":960692224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2598703104,"data_stored":2598699032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1711013888,"data_stored":1711013888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2583695360,"data_stored":2583691288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1312423936,"data_stored":1312423936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:46:04.209 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T21:46:04.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 36 2024-03-26T21:46:04.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:04.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:04.353 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:04.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:04.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.615 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:04.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:04.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:04.742 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:04.844 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:04.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:04.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:04.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:05.011 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:05.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:05.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:05.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:05.278 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:05.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:05.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:05 smithi067 ceph-mon[33346]: pgmap v811: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 170 op/s; 1272/212784 objects degraded (0.598%); 6825/212784 objects misplaced (3.207%); 11 MiB/s, 207 objects/s recovering 2024-03-26T21:46:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:05 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:46:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:05 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:46:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:05 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:46:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:05 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:46:05.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:05.547 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[29406]: pgmap v811: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 170 op/s; 1272/212784 objects degraded (0.598%); 6825/212784 objects misplaced (3.207%); 11 MiB/s, 207 objects/s recovering 2024-03-26T21:46:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:46:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:46:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:46:05.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:46:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[40587]: pgmap v811: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 170 op/s; 1272/212784 objects degraded (0.598%); 6825/212784 objects misplaced (3.207%); 11 MiB/s, 207 objects/s recovering 2024-03-26T21:46:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:46:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:46:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:46:05.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:05 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:46:05.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:05.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:46:05.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:05.811 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:05.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:05.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:06.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:06.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:06.175 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:06.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:06.309 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:06.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:06.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:46:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:06 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:46:06.507 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:139902 10.15 10:ae86aae8:::benchmark_data_smithi055_163313_object35853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:05.924092+0000", 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.0071069180000000003, 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0071460869999999998, 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 139902 2024-03-26T21:46:06.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924092+0000", 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924092+0000", 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:46:06.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924094+0000", 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1569999999999998e-06 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924103+0000", 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5839999999999995e-06 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924118+0000", 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5291e-05 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924129+0000", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1148999999999999e-05 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924217+0000", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7477000000000003e-05 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.924462+0000", 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00024553500000000001 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.925168+0000", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00070564499999999997 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.928060+0000", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028923600000000001 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931232+0000", 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0031713700000000002 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.512 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.15609.0:139902 10.15 e386/375 ondisk, result = 0)", 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:05.931046+0000", 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.000153554, 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029413799999999999, 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:46:06.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931046+0000", 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931046+0000", 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931046+0000", 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4300000000000002e-07 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931053+0000", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5300000000000002e-06 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931068+0000", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4883e-05 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931076+0000", 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4800000000000001e-06 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931261+0000", 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018526799999999999 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931268+0000", 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1879999999999999e-06 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "MOSDPGPush(10.18 386/364 [PushOp(10:1de13079:::benchmark_data_smithi055_163313_object78914:head, version: 358'4164, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:1de13079:::benchmark_data_smithi055_163313_object78914:head@358'4164, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:05.931282+0000", 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 4294967295.9999166, 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015023400000000001, 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "queued for pg", 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931282+0000", 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931282+0000", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931283+0000", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9999999999999997e-07 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931319+0000", 2024-03-26T21:46:06.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6545999999999999e-05 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931337+0000", 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7980999999999999e-05 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:05.931354+0000", 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7008999999999999e-05 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:06.519 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.520 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:06.520 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:06.520 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 3 2024-03-26T21:46:06.520 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:46:06.520 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:46:06.520 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:46:06.520 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:46:06.520 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:46:06.520 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:46:06.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:06 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:46:06.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.600 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:06.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:06.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:06.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:06.869 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:06.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:06.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:07.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:07.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:07.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:07.461 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:07 smithi067 ceph-mon[33346]: pgmap v812: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 411 op/s; 1272/212241 objects degraded (0.599%); 6825/212241 objects misplaced (3.216%); 15 MiB/s, 264 objects/s recovering 2024-03-26T21:46:07.462 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:07 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1272/212241 objects degraded (0.599%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:07.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:07 smithi055 ceph-mon[40587]: pgmap v812: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 411 op/s; 1272/212241 objects degraded (0.599%); 6825/212241 objects misplaced (3.216%); 15 MiB/s, 264 objects/s recovering 2024-03-26T21:46:07.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:07 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1272/212241 objects degraded (0.599%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:07 smithi055 ceph-mon[29406]: pgmap v812: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 411 op/s; 1272/212241 objects degraded (0.599%); 6825/212241 objects misplaced (3.216%); 15 MiB/s, 264 objects/s recovering 2024-03-26T21:46:07.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:07 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1272/212241 objects degraded (0.599%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:07.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:07.669 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:07.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:07.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:07.794 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:07.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:08.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:08.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:08.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:08.200 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:08.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:08.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:08.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:08.472 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:08 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:46:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:08 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:46:08.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:08 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:46:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:08 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:46:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:08 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:46:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:08 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:46:08.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:08.609 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:08.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:08.742 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:08.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:08.875 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:08.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:08.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:09.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:09.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:09.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:09.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[40587]: pgmap v813: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 568 op/s; 1272/205539 objects degraded (0.619%); 4061/205539 objects misplaced (1.976%); 32 MiB/s, 547 objects/s recovering 2024-03-26T21:46:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[40587]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[29406]: pgmap v813: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 568 op/s; 1272/205539 objects degraded (0.619%); 4061/205539 objects misplaced (1.976%); 32 MiB/s, 547 objects/s recovering 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:09 smithi055 ceph-mon[29406]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:46:09.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:09 smithi067 ceph-mon[33346]: pgmap v813: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 568 op/s; 1272/205539 objects degraded (0.619%); 4061/205539 objects misplaced (1.976%); 32 MiB/s, 547 objects/s recovering 2024-03-26T21:46:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:09 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:09 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:09 smithi067 ceph-mon[33346]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T21:46:09.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:09.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:09.660 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:09.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:09.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:09.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:09.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:09.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:10.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:10.055 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:10.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:10.185 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:10.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:10.315 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:10.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:10 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:46:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:10 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:46:10.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:10.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:10 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:46:10.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:10 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:46:10.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:10 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:46:10.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:10 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:46:10.576 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:10.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:10.703 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:10.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.830 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:10.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:10.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:10.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:11.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:11.091 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:11.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:11.223 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:11.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:11.357 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:11.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:11 smithi067 ceph-mon[33346]: pgmap v815: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 602 op/s; 1272/202902 objects degraded (0.627%); 4061/202902 objects misplaced (2.001%); 29 MiB/s, 490 objects/s recovering 2024-03-26T21:46:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:11 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:11 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.494 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:11.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[40587]: pgmap v815: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 602 op/s; 1272/202902 objects degraded (0.627%); 4061/202902 objects misplaced (2.001%); 29 MiB/s, 490 objects/s recovering 2024-03-26T21:46:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[29406]: pgmap v815: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 602 op/s; 1272/202902 objects degraded (0.627%); 4061/202902 objects misplaced (2.001%); 29 MiB/s, 490 objects/s recovering 2024-03-26T21:46:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:11 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:11.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:11.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:11.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:11.764 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:11.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:11.892 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:11.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:11.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:12.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:12.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:12.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:12.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:12.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:46:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:46:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:46:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1272/202902 objects degraded (0.627%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:12 smithi067 ceph-mon[33346]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:46:12.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:12.525 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1272/202902 objects degraded (0.627%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:12.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[29406]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1272/202902 objects degraded (0.627%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:46:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:12 smithi055 ceph-mon[40587]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T21:46:12.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.654 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:12.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.780 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:12.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:12.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:13.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:13.063 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:13.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:13.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:13.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:13.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: pgmap v817: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 467 op/s; 1272/201567 objects degraded (0.631%); 3876/201567 objects misplaced (1.923%); 21 MiB/s, 331 objects/s recovering 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: 10.14 scrub starts 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: 10.c scrub starts 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: 10.14 scrub ok 2024-03-26T21:46:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:13 smithi067 ceph-mon[33346]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: pgmap v817: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 467 op/s; 1272/201567 objects degraded (0.631%); 3876/201567 objects misplaced (1.923%); 21 MiB/s, 331 objects/s recovering 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: 10.14 scrub starts 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: 10.c scrub starts 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: 10.14 scrub ok 2024-03-26T21:46:13.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[29406]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: pgmap v817: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 467 op/s; 1272/201567 objects degraded (0.631%); 3876/201567 objects misplaced (1.923%); 21 MiB/s, 331 objects/s recovering 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: 10.14 scrub starts 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: 10.c scrub starts 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: 10.14 scrub ok 2024-03-26T21:46:13.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:13 smithi055 ceph-mon[40587]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T21:46:13.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:13.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:13.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:13.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:13.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:13.927 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:13.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:14.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:14.041 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:14.042 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:14.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:46:14.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:14.187 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:14.226 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:46:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:46:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[40587]: 10.13 scrub starts 2024-03-26T21:46:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[40587]: 10.c scrub ok 2024-03-26T21:46:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:46:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:46:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[29406]: 10.13 scrub starts 2024-03-26T21:46:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:14 smithi055 ceph-mon[29406]: 10.c scrub ok 2024-03-26T21:46:14.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:14.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:14 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:46:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:14 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:46:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:14 smithi067 ceph-mon[33346]: 10.13 scrub starts 2024-03-26T21:46:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:14 smithi067 ceph-mon[33346]: 10.c scrub ok 2024-03-26T21:46:14.489 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:14.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.627 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:14.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:14.759 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:14.861 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:14.889 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:14.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:14.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:15.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:15.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:15.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:15.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:15.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:15.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.410 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:15.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 10.18 scrub starts 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: pgmap v820: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 520 op/s; 418/195444 objects degraded (0.214%); 2115/195444 objects misplaced (1.082%); 21 MiB/s, 486 objects/s recovering 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 10.13 scrub ok 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 10.18 scrub ok 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 10.19 scrub starts 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:46:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:15 smithi067 ceph-mon[33346]: 10.19 scrub ok 2024-03-26T21:46:15.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.535 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 10.18 scrub starts 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: pgmap v820: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 520 op/s; 418/195444 objects degraded (0.214%); 2115/195444 objects misplaced (1.082%); 21 MiB/s, 486 objects/s recovering 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 10.13 scrub ok 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 10.18 scrub ok 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 10.19 scrub starts 2024-03-26T21:46:15.541 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[40587]: 10.19 scrub ok 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 10.18 scrub starts 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: pgmap v820: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 520 op/s; 418/195444 objects degraded (0.214%); 2115/195444 objects misplaced (1.082%); 21 MiB/s, 486 objects/s recovering 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 10.13 scrub ok 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 10.18 scrub ok 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 10.19 scrub starts 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:46:15.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:15 smithi055 ceph-mon[29406]: 10.19 scrub ok 2024-03-26T21:46:15.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:15.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:15.792 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:15.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:15.918 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:15.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:16.020 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:16.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:16.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:16.220 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:46:16.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:16.409 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:16.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:16 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:46:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:16 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:46:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:16 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:46:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:16 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:46:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:16 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:16.540 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:16.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:46:16.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:16 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:16.571 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:16.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:16.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:16.806 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:16.832 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:16.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:16.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:16.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:17.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:17.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:17.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:17.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:17.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:17.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:17.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: pgmap v821: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 450 op/s; 418/194808 objects degraded (0.215%); 2115/194808 objects misplaced (1.086%); 19 MiB/s, 446 objects/s recovering 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:46:17.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 418/194808 objects degraded (0.215%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: pgmap v821: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 450 op/s; 418/194808 objects degraded (0.215%); 2115/194808 objects misplaced (1.086%); 19 MiB/s, 446 objects/s recovering 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:46:17.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:46:17.468 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:46:17.468 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:46:17.468 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:17 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 418/194808 objects degraded (0.215%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:17.473 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: pgmap v821: 158 pgs: 1 peering, 2 active+clean+scrubbing, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 450 op/s; 418/194808 objects degraded (0.215%); 2115/194808 objects misplaced (1.086%); 19 MiB/s, 446 objects/s recovering 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/22297545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:46:17.474 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 418/194808 objects degraded (0.215%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:17.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:17.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:17.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:17.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:17.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:17.749 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:17.850 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:17.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:17.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:17.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:18.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:18.105 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:18.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:18.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:18.263 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:18.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:18.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:18.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:18.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:18.651 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:18.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:18 smithi067 ceph-mon[33346]: pgmap v823: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 155 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 18 op/s; 418/187458 objects degraded (0.223%); 1924/187458 objects misplaced (1.026%) 2024-03-26T21:46:18.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:18.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[29406]: pgmap v823: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 155 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 18 op/s; 418/187458 objects degraded (0.223%); 1924/187458 objects misplaced (1.026%) 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:18.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-mon[40587]: pgmap v823: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 155 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 18 op/s; 418/187458 objects degraded (0.223%); 1924/187458 objects misplaced (1.026%) 2024-03-26T21:46:18.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:18.906 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:18.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:19.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:19.136 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:19.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:19.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:19.297 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:19.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:19.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:19.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:19.652 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:19.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:19.700 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:19.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:19.829 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:19.874 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:19 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:46:19.874 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:19 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:46:19.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:19 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:46:19.874 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:19 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:46:19.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:19.962 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:19.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:19 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:46:19.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:19 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:46:20.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:20.096 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:20.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:20.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:20.357 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:20.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:20.485 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:20.588 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:20.616 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:20.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:20.845 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:20.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:20.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:20.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:20.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:20 smithi067 ceph-mon[33346]: pgmap v824: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/184596 objects degraded (0.226%); 1924/184596 objects misplaced (1.042%) 2024-03-26T21:46:21.004 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:21.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:20 smithi055 ceph-mon[29406]: pgmap v824: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/184596 objects degraded (0.226%); 1924/184596 objects misplaced (1.042%) 2024-03-26T21:46:21.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:20 smithi055 ceph-mon[40587]: pgmap v824: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/184596 objects degraded (0.226%); 1924/184596 objects misplaced (1.042%) 2024-03-26T21:46:21.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:46:21.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:21.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.269 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:21.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:21.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.396 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:21.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:21.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:21.521 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:21.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:21.648 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 10 pg_num to 36 2024-03-26T21:46:21.655 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:21.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:21.788 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:21.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:46:21.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:46:21.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:46:21.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2073684932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2073684932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:21.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:21 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:21.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:21.936 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:21.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2073684932' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:21.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:21 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-26T21:46:22.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:22.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:22.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:22.195 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:22.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:22.323 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:22.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:22.457 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:22.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:22.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:22.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:22.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:22.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: pgmap v826: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 418/183087 objects degraded (0.228%); 1712/183087 objects misplaced (0.935%) 2024-03-26T21:46:22.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 418/184596 objects degraded (0.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:46:22.995 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: pgmap v826: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 418/183087 objects degraded (0.228%); 1712/183087 objects misplaced (0.935%) 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 418/184596 objects degraded (0.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: pgmap v826: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 418/183087 objects degraded (0.228%); 1712/183087 objects misplaced (0.935%) 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:22.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 418/184596 objects degraded (0.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:23.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.048 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:23.142 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:46:23.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:23.162 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:46:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:23.182 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:46:23.202 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:46:23.223 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:46:23.243 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:46:23.263 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:46:23.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:23.284 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:46:23.285 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:46:23.286 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:46:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:23.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:23.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:23.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:23.641 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:46:23.641 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:46:23.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:23.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:23.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:23.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:23.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: 3.0 deep-scrub starts 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: 3.0 deep-scrub ok 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:46:23.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3836807652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:23.996 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: 3.0 deep-scrub starts 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: 3.0 deep-scrub ok 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3836807652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: 3.0 deep-scrub starts 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: 3.0 deep-scrub ok 2024-03-26T21:46:23.997 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T21:46:23.998 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3836807652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:24.063 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:24.097 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:24.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:24.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:24.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:24.381 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:24.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:24.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:24.525 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:24.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:24.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:24.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:24.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:24.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:24.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: pgmap v828: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/175689 objects degraded (0.238%); 1712/175689 objects misplaced (0.974%) 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:46:24.963 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: pgmap v828: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/175689 objects degraded (0.238%); 1712/175689 objects misplaced (0.974%) 2024-03-26T21:46:24.964 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.964 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:46:24.964 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:24 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: pgmap v828: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 418/175689 objects degraded (0.238%); 1712/175689 objects misplaced (0.974%) 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T21:46:24.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:24 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-26T21:46:24.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:46:25.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:25.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:25.168 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:25.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:25.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:25.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:25.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:25.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.551 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:25.653 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:25.678 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:25.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:25.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:25.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:25.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 6.d deep-scrub starts 2024-03-26T21:46:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 10.3 scrub starts 2024-03-26T21:46:25.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 6.d deep-scrub ok 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 10.3 scrub ok 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 10.2 deep-scrub starts 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 10.c scrub starts 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: 10.c scrub ok 2024-03-26T21:46:25.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:25 smithi067 ceph-mon[33346]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:46:26.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 6.d deep-scrub starts 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 10.3 scrub starts 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 6.d deep-scrub ok 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 10.3 scrub ok 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 10.2 deep-scrub starts 2024-03-26T21:46:26.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 10.c scrub starts 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: 10.c scrub ok 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[40587]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 6.d deep-scrub starts 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 10.3 scrub starts 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 6.d deep-scrub ok 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 10.3 scrub ok 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 10.2 deep-scrub starts 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 10.c scrub starts 2024-03-26T21:46:26.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: 10.c scrub ok 2024-03-26T21:46:26.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:25 smithi055 ceph-mon[29406]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T21:46:26.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:26.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:26.250 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:26.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:26.311 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:26.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.437 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:26.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:26.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:46:26.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:26.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:26.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:26.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:26.769 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[29406]: 3.3 deep-scrub starts 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[29406]: 3.3 deep-scrub ok 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[29406]: pgmap v831: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 652 op/s; 418/175206 objects degraded (0.239%); 1712/175206 objects misplaced (0.977%) 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[29406]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[40587]: 3.3 deep-scrub starts 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[40587]: 3.3 deep-scrub ok 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[40587]: pgmap v831: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 652 op/s; 418/175206 objects degraded (0.239%); 1712/175206 objects misplaced (0.977%) 2024-03-26T21:46:26.770 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:26 smithi055 ceph-mon[40587]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:46:26.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:26.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:26.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:26.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:26 smithi067 ceph-mon[33346]: 3.3 deep-scrub starts 2024-03-26T21:46:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:26 smithi067 ceph-mon[33346]: 3.3 deep-scrub ok 2024-03-26T21:46:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:26 smithi067 ceph-mon[33346]: pgmap v831: 168 pgs: 2 peering, 10 unknown, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 652 op/s; 418/175206 objects degraded (0.239%); 1712/175206 objects misplaced (0.977%) 2024-03-26T21:46:26.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:26 smithi067 ceph-mon[33346]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T21:46:27.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:27.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:27.262 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:27.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:27.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:27.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:27.547 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:27.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:27.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:27.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.2 deep-scrub ok 2024-03-26T21:46:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:46:27.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 394'6316 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 394'9480 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.23 starting backfill to osd.4 from (0'0,362'4450] MIN to 394'6276 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 418/175206 objects degraded (0.239%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:27.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:46:27.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:46:27.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:27.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:27.882 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:46:27.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:46:27.883 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T21:46:27.883 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T21:46:27.884 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:46:27.933 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.2 deep-scrub ok 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 394'6316 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 394'9480 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.23 starting backfill to osd.4 from (0'0,362'4450] MIN to 394'6276 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:46:27.934 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 418/175206 objects degraded (0.239%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.2 deep-scrub ok 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:46:27.935 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 394'6316 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 394'9480 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.23 starting backfill to osd.4 from (0'0,362'4450] MIN to 394'6276 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 394'9271 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 394'9376 2024-03-26T21:46:27.936 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 394'6449 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 394'9436 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 394'9474 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 418/175206 objects degraded (0.239%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T21:46:27.937 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:46:27.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:27.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:28.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:28.119 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:28.221 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:28.264 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:28.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:28.405 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:28.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:28.539 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:28.558 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:28.565 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:28.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:28.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:28.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:28.868 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:28.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:28.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 peering, 4 unknown, 156 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 419/162870 objects degraded (0.257%); 10026/162870 objects misplaced (6.156%) 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:46:28.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[40587]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 peering, 4 unknown, 156 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 419/162870 objects degraded (0.257%); 10026/162870 objects misplaced (6.156%) 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-mon[29406]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 peering, 4 unknown, 156 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 419/162870 objects degraded (0.257%); 10026/162870 objects misplaced (6.156%) 2024-03-26T21:46:28.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2130854779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:46:28.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:28 smithi067 ceph-mon[33346]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T21:46:29.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:29.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:29.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:29.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:29.256 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:29.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:29.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:29.431 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:29.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:29.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:29.582 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:46:29.582 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:46:29.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:29.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:29.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:29.930 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:29.962 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:29.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:30.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:30.096 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:30.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:30.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:30.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:30.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:30.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:30.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:30.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:30.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.782 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:30.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:30.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:30.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:31.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[40587]: pgmap v837: 168 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 419/167760 objects degraded (0.250%); 21759/167760 objects misplaced (12.970%) 2024-03-26T21:46:31.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[40587]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:46:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[29406]: pgmap v837: 168 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 419/167760 objects degraded (0.250%); 21759/167760 objects misplaced (12.970%) 2024-03-26T21:46:31.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:30 smithi055 ceph-mon[29406]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:46:31.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.187 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:30 smithi067 ceph-mon[33346]: pgmap v837: 168 pgs: 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 419/167760 objects degraded (0.250%); 21759/167760 objects misplaced (12.970%) 2024-03-26T21:46:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:30 smithi067 ceph-mon[33346]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T21:46:31.289 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:31.322 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:31.424 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:31.456 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:31.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:31.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:31.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:31.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:31.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:46:31.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:31.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:31.987 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:31.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:32.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:32.123 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:32.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:32.371 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:32.382 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:46:32.407 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:32.422 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:46:32.453 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:46:32.484 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:46:32.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:32.517 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:46:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:32.549 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:46:32.584 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:46:32.623 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:46:32.632 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:46:32.633 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:46:32.633 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:46:32.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:32.678 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:32.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:32.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[29406]: pgmap v839: 168 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 824 B/s rd, 0 op/s; 419/165756 objects degraded (0.253%); 23279/165756 objects misplaced (14.044%) 2024-03-26T21:46:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 419/165756 objects degraded (0.253%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:32.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/68311194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[40587]: pgmap v839: 168 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 824 B/s rd, 0 op/s; 419/165756 objects degraded (0.253%); 23279/165756 objects misplaced (14.044%) 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 419/165756 objects degraded (0.253%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/68311194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:32.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:32.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:32.937 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:32.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:32 smithi067 ceph-mon[33346]: pgmap v839: 168 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 824 B/s rd, 0 op/s; 419/165756 objects degraded (0.253%); 23279/165756 objects misplaced (14.044%) 2024-03-26T21:46:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 419/165756 objects degraded (0.253%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:32 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/68311194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:32.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:33.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:33.069 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:33.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:46:33.171 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.223 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:33.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:33.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:33.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:33.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:33.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:33.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:33.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:33.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:33.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:33.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:34.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:34.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:34.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:34.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:34.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:34.610 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:34.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.749 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:159160 10.6 10:62bb2f9b:::benchmark_data_smithi055_163313_object55111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:34.819751+0000", 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0092692910000000007, 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093064259999999996, 2024-03-26T21:46:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 159160 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819751+0000", 2024-03-26T21:46:34.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819751+0000", 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819753+0000", 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7710000000000001e-06 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:34.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819759+0000", 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6999999999999996e-06 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819768+0000", 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3570000000000004e-06 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819782+0000", 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3726999999999999e-05 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819854+0000", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1670999999999999e-05 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.819985+0000", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013135800000000001 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,4", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.820413+0000", 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042857099999999999 2024-03-26T21:46:34.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.825496+0000", 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050824989999999999 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.827114+0000", 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016183199999999999 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15609.0:159169 10.1a e400/393 10:5d03fe30:::benchmark_data_smithi055_163313_object55120:head v 400'9922, mlcod=400'9920)", 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:34.824627+0000", 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0043931650000000001, 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045373929999999998, 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824627+0000", 2024-03-26T21:46:34.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824627+0000", 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824628+0000", 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0640000000000001e-06 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:34.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824636+0000", 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1369999999999996e-06 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824649+0000", 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3220999999999999e-05 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824657+0000", 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6909999999999997e-06 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824754+0000", 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7015999999999994e-05 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.824773+0000", 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9307e-05 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.15609.0:159166 10.12 e400/364 10:4846705c:::benchmark_data_smithi055_163313_object55117:head v 400'6719, mlcod=400'6717)", 2024-03-26T21:46:34.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:34.826420+0000", 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0026008670000000002, 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028134620000000001, 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826420+0000", 2024-03-26T21:46:34.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826420+0000", 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826420+0000", 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3099999999999997e-07 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:34.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826427+0000", 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5100000000000004e-06 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826436+0000", 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3729999999999999e-06 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826444+0000", 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8299999999999996e-06 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826542+0000", 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.815e-05 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.826562+0000", 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9768000000000001e-05 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:159173 10.9 10:902b5d58:::benchmark_data_smithi055_163313_object55124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-26T21:46:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:34.828953+0000", 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.7893999999999997e-05, 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034793699999999999, 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 159173 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828953+0000", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828953+0000", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828954+0000", 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7519999999999999e-06 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828961+0000", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0299999999999996e-06 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828978+0000", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6654e-05 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.828987+0000", 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0289999999999994e-06 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.829083+0000", 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5733e-05 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:34.829210+0000", 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012674600000000001 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 4 2024-03-26T21:46:34.840 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:46:34.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:34.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:34.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:34.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:34 smithi067 ceph-mon[33346]: pgmap v840: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.94k op/s; 419/156069 objects degraded (0.268%); 20502/156069 objects misplaced (13.136%); 27 MiB/s, 413 objects/s recovering 2024-03-26T21:46:34.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:35.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:35.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:34 smithi055 ceph-mon[40587]: pgmap v840: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.94k op/s; 419/156069 objects degraded (0.268%); 20502/156069 objects misplaced (13.136%); 27 MiB/s, 413 objects/s recovering 2024-03-26T21:46:35.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:34 smithi055 ceph-mon[29406]: pgmap v840: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 52 MiB/s wr, 1.94k op/s; 419/156069 objects degraded (0.268%); 20502/156069 objects misplaced (13.136%); 27 MiB/s, 413 objects/s recovering 2024-03-26T21:46:35.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:35.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:35.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:35.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:35.350 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:35.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:35.474 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:35.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:35.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:35.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:35.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:35.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:46:35.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:35.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:35.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:36.017 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:36.058 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:36.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:36.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:46:36.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:36.270 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:36.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:36.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:36.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:36.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:36.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:36.725 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "epoch": 401, 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "created": "2024-03-26T21:22:56.673257+0000", 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "modified": "2024-03-26T21:46:35.758692+0000", 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "last_up_change": "2024-03-26T21:41:24.572964+0000", 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "last_in_change": "2024-03-26T21:44:48.291896+0000", 2024-03-26T21:46:36.730 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_num": 5801984, 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_set": [ 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "noscrub", 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "pglog_hardlimit", 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "purged_snapdirs", 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_deletes", 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "sortbitwise" 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_version": 20, 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T21:46:36.731 INFO:teuthology.orchestra.run.smithi055.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_snapshot": "", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_max": 10, 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "max_osd": 8, 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "require_min_compat_client": "luminous", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "min_compat_client": "luminous", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "require_osd_release": "squid", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "allow_crimson": false, 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "pools": [ 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 1, 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".mgr", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:26:13.884991+0000", 2024-03-26T21:46:36.732 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.733 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 1, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 1, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 1, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 1, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 1, 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.734 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "265", 2024-03-26T21:46:36.735 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.736 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.736 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.737 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.738 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.739 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.740 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.741 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:46:36.742 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_max": 32, 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_min": 1 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": {} 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.743 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.744 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.744 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 7.8899998664855957, 2024-03-26T21:46:36.744 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 7.8899998664855957, 2024-03-26T21:46:36.744 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T21:46:36.744 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 3, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 3, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 2, 2024-03-26T21:46:36.745 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "datapool", 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.142617+0000", 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 8193, 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.746 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.747 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 3, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 3, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 3, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 3, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 3, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.748 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "71", 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 2, 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 71, 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.749 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.750 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.751 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.752 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.753 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd": {} 2024-03-26T21:46:36.754 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.754 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.754 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.754 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.6500000953674316, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.6500000953674316, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.755 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 3, 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".rgw.root", 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.591599+0000", 2024-03-26T21:46:36.756 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.757 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:46:36.758 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.759 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "68", 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.760 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.761 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.762 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.762 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.762 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.762 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.762 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.763 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:46:36.764 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.5, 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.5, 2024-03-26T21:46:36.765 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5, 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5, 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.766 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 4, 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.log", 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:34.904126+0000", 2024-03-26T21:46:36.767 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.768 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:46:36.769 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.770 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "70", 2024-03-26T21:46:36.771 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.772 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.773 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.774 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.774 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.774 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.774 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.774 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.775 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.776 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.777 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.25, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.25, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.25, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.25, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.778 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.779 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.779 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.779 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 5, 2024-03-26T21:46:36.779 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.control", 2024-03-26T21:46:36.779 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:36.852373+0000", 2024-03-26T21:46:36.780 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.780 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.780 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.780 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.780 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.781 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.782 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "72", 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.783 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.784 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.785 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.25, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.25, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.25, 2024-03-26T21:46:36.786 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.25, 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 6, 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.meta", 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:38.937273+0000", 2024-03-26T21:46:36.787 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:36.789 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.789 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.789 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.789 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.789 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.790 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.791 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "74", 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.792 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.793 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.794 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.795 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_bias": 4 2024-03-26T21:46:36.796 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.75, 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.75, 2024-03-26T21:46:36.797 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 10, 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "unique_pool_3", 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:43:27.602172+0000", 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:46:36.799 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 36, 2024-03-26T21:46:36.800 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 36, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 36, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 36, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 36, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:46:36.801 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "395", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "393", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:46:36.802 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:46:36.803 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:46:36.804 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:46:36.805 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:46:36.806 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "rados": {} 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2, 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.2200000286102295, 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:46:36.807 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.2200000286102295, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 1, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 1 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae", 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.808 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 8, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 396, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6801", 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.809 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6802", 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.810 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6804", 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6803", 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6801/317381212", 2024-03-26T21:46:36.811 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6802/317381212", 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6804/317381212", 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6803/317381212", 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db", 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.812 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 13, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 396, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6805", 2024-03-26T21:46:36.813 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6806", 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.814 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6808", 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6807", 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:46:36.815 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6805/3733554168", 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6806/3733554168", 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6808/3733554168", 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6807/3733554168", 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.816 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b", 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 18, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 396, 2024-03-26T21:46:36.817 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6809", 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.818 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6810", 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6812", 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:46:36.819 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6811", 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6809/3059242923", 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6810/3059242923", 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6812/3059242923", 2024-03-26T21:46:36.820 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6811/3059242923", 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b", 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.821 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 26, 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 395, 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6813", 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.822 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6814", 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6816", 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.823 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.824 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6815", 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6813/3851321468", 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6814/3851321468", 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6816/3851321468", 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6815/3851321468", 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.825 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598", 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 31, 2024-03-26T21:46:36.826 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 92, 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 133, 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 396, 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 93, 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6800", 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.827 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6801", 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.828 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6803", 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.829 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6802", 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.831 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6800/254752278", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6801/254752278", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6803/254752278", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6802/254752278", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213", 2024-03-26T21:46:36.832 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 41, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 395, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6804", 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6805", 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6807", 2024-03-26T21:46:36.835 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6806", 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6804/278711358", 2024-03-26T21:46:36.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6805/278711358", 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6807/278711358", 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6806/278711358", 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9", 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.837 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 50, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 395, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6808", 2024-03-26T21:46:36.838 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6809", 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.839 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6811", 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6810", 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:46:36.840 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6808/3926251639", 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6809/3926251639", 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6811/3926251639", 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6810/3926251639", 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.841 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55", 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 59, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 248, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 276, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 395, 2024-03-26T21:46:36.842 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 249, 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6812", 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.843 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6813", 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6815", 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.844 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6814", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6812/650373445", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6813/650373445", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6815/650373445", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6814/650373445", 2024-03-26T21:46:36.845 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_xinfo": [ 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.846 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:45:28.485629+0000", 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:46:10.509776+0000", 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.847 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:46:10.895502+0000", 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.848 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:46:08.115122+0000", 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:46:35.009225+0000", 2024-03-26T21:46:36.849 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 92 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:45:58.974597+0000", 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:46:36.850 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:46:29.519748+0000", 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:46:36.851 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:45:34.275708+0000", 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 248 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap": [], 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_items": [ 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "3.3", 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 4, 2024-03-26T21:46:36.852 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 2 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.4", 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.853 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.6", 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.854 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.17", 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 2 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.18", 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:46:36.855 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 0 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_primaries": [], 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_temp": [ 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.17", 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: 6 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.856 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.1b", 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 1, 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 6 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.1c", 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: 7 2024-03-26T21:46:36.857 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.1d", 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: 6, 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: 0 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.1e", 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: 2, 2024-03-26T21:46:36.858 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: 6 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.1f", 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: 2, 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: 3, 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: 6 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.859 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.20", 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: 1, 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: 0 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.21", 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: 7 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.22", 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: 0, 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: 1, 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: 4 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "10.23", 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: 2, 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: 5 2024-03-26T21:46:36.861 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_temp": [], 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklist": { 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2649740776": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/4141219490": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/147833426": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2854772543": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/2804789175": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2199694469": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1614234257": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2834138692": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1740219825": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/268664696": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:46:36.862 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/536717527": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/966571165": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2757111431": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/559658018": "2024-03-27T21:23:57.787077+0000" 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "range_blocklist": {}, 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profiles": { 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "default": { 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "crush-failure-domain": "osd", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "k": "2", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "m": "1", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin": "jerasure", 2024-03-26T21:46:36.863 INFO:teuthology.orchestra.run.smithi055.stdout: "technique": "reed_sol_van" 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps_queue": [], 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "new_removed_snaps": [], 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "new_purged_snaps": [], 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_node_flags": {}, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "device_class_flags": {}, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode": { 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_enabled": false, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_bucket_count": 0, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "degraded_stretch_mode": 0, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "recovering_stretch_mode": 0, 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_bucket": 0 2024-03-26T21:46:36.864 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:46:36.865 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:46:36.865 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:36 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:36.866 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:36 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:36.866 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:36 smithi067 ceph-mon[33346]: pgmap v841: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 419/154356 objects degraded (0.271%); 19717/154356 objects misplaced (12.774%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:46:36.866 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:36 smithi067 ceph-mon[33346]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:46:36.866 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1224784131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:46:36.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:36.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:37.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[40587]: pgmap v841: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 419/154356 objects degraded (0.271%); 19717/154356 objects misplaced (12.774%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:46:37.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[40587]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1224784131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[29406]: pgmap v841: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 419/154356 objects degraded (0.271%); 19717/154356 objects misplaced (12.774%); 16 MiB/s, 250 objects/s recovering 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[29406]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T21:46:37.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1224784131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:46:37.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:37.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.319 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:37.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:37.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:37.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:37.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:37.586 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:37.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:37.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:37.811 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:37.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.856 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:37.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:37.920 INFO:tasks.thrashosds.thrasher:j is {'epoch': 401, 'fsid': 'e5f52b02-ebb6-11ee-95d0-87774f69a715', 'created': '2024-03-26T21:22:56.673257+0000', 'modified': '2024-03-26T21:46:35.758692+0000', 'last_up_change': '2024-03-26T21:41:24.572964+0000', 'last_in_change': '2024-03-26T21:44:48.291896+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T21:28:33.142617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T21:28:33.591599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T21:28:34.904126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T21:28:36.852373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T21:28:38.937273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-26T21:43:27.602172+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '395', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '393', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2.2200000286102295, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2.2200000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4f487071-c151-406b-b25b-3135cb3c63ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 396, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6801', 'nonce': 317381212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6802', 'nonce': 317381212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6804', 'nonce': 317381212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6803', 'nonce': 317381212}]}, 'public_addr': '172.21.15.55:6801/317381212', 'cluster_addr': '172.21.15.55:6802/317381212', 'heartbeat_back_addr': '172.21.15.55:6804/317381212', 'heartbeat_front_addr': '172.21.15.55:6803/317381212', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 396, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6805', 'nonce': 3733554168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6806', 'nonce': 3733554168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6808', 'nonce': 3733554168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6807', 'nonce': 3733554168}]}, 'public_addr': '172.21.15.55:6805/3733554168', 'cluster_addr': '172.21.15.55:6806/3733554168', 'heartbeat_back_addr': '172.21.15.55:6808/3733554168', 'heartbeat_front_addr': '172.21.15.55:6807/3733554168', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 396, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6809', 'nonce': 3059242923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6810', 'nonce': 3059242923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6812', 'nonce': 3059242923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6811', 'nonce': 3059242923}]}, 'public_addr': '172.21.15.55:6809/3059242923', 'cluster_addr': '172.21.15.55:6810/3059242923', 'heartbeat_back_addr': '172.21.15.55:6812/3059242923', 'heartbeat_front_addr': '172.21.15.55:6811/3059242923', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '30637a8d-5c1b-4ede-8214-a4595c875d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6813', 'nonce': 3851321468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6814', 'nonce': 3851321468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6816', 'nonce': 3851321468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6815', 'nonce': 3851321468}]}, 'public_addr': '172.21.15.55:6813/3851321468', 'cluster_addr': '172.21.15.55:6814/3851321468', 'heartbeat_back_addr': '172.21.15.55:6816/3851321468', 'heartbeat_front_addr': '172.21.15.55:6815/3851321468', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'cd2d68be-4b10-4242-826b-e085c2ce5598', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 92, 'up_from': 133, 'up_thru': 396, 'down_at': 93, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 254752278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6801', 'nonce': 254752278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 254752278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 254752278}]}, 'public_addr': '172.21.15.67:6800/254752278', 'cluster_addr': '172.21.15.67:6801/254752278', 'heartbeat_back_addr': '172.21.15.67:6803/254752278', 'heartbeat_front_addr': '172.21.15.67:6802/254752278', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 278711358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6805', 'nonce': 278711358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 278711358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 278711358}]}, 'public_addr': '172.21.15.67:6804/278711358', 'cluster_addr': '172.21.15.67:6805/278711358', 'heartbeat_back_addr': '172.21.15.67:6807/278711358', 'heartbeat_front_addr': '172.21.15.67:6806/278711358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd47c79f3-96bf-46b6-a7f3-a315446d04b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 395, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 3926251639}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 3926251639}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 3926251639}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 3926251639}]}, 'public_addr': '172.21.15.67:6808/3926251639', 'cluster_addr': '172.21.15.67:6809/3926251639', 'heartbeat_back_addr': '172.21.15.67:6811/3926251639', 'heartbeat_front_addr': '172.21.15.67:6810/3926251639', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f10b98b8-887d-4f83-8bc9-99f5c302dc55', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 59, 'last_clean_end': 248, 'up_from': 276, 'up_thru': 395, 'down_at': 249, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 650373445}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6813', 'nonce': 650373445}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 650373445}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 650373445}]}, 'public_addr': '172.21.15.67:6812/650373445', 'cluster_addr': '172.21.15.67:6813/650373445', 'heartbeat_back_addr': '172.21.15.67:6815/650373445', 'heartbeat_front_addr': '172.21.15.67:6814/650373445', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:45:28.485629+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:46:10.509776+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:46:10.895502+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:46:08.115122+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:46:35.009225+0000', 'dead_epoch': 92}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:45:58.974597+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:46:29.519748+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:45:34.275708+0000', 'dead_epoch': 248}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.17', 'osds': [4, 6]}, {'pgid': '10.1b', 'osds': [0, 1, 6]}, {'pgid': '10.1c', 'osds': [0, 4, 7]}, {'pgid': '10.1d', 'osds': [4, 6, 0]}, {'pgid': '10.1e', 'osds': [2, 4, 6]}, {'pgid': '10.1f', 'osds': [2, 3, 6]}, {'pgid': '10.20', 'osds': [1, 0]}, {'pgid': '10.21', 'osds': [4, 0, 7]}, {'pgid': '10.22', 'osds': [0, 1, 4]}, {'pgid': '10.23', 'osds': [2, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.55:0/2649740776': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/4141219490': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/147833426': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2854772543': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/2804789175': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/2199694469': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/1614234257': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/2834138692': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/1740219825': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/268664696': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:6800/536717527': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/966571165': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2757111431': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/559658018': '2024-03-27T21:23:57.787077+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T21:46:37.920 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-26T21:46:37.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:37.991 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:37.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:38.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[40587]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:46:38.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:38.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 419/154356 objects degraded (0.271%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[29406]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:46:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:38.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 419/154356 objects degraded (0.271%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:38.091 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:37 smithi067 ceph-mon[33346]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T21:46:38.091 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:37 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:38.091 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 419/154356 objects degraded (0.271%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:38.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:38.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:38.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:38.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:38.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:38.426 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:38.430 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:38.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:38.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:38.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:38.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:38.790 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:46:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:46:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:46:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:46:38.791 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:46:38.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:38.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:38.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:38.959 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:38.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[29406]: pgmap v844: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 990 op/s; 418/144210 objects degraded (0.290%); 18262/144210 objects misplaced (12.663%); 25 MiB/s, 490 objects/s recovering 2024-03-26T21:46:38.990 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[29406]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[40587]: pgmap v844: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 990 op/s; 418/144210 objects degraded (0.290%); 18262/144210 objects misplaced (12.663%); 25 MiB/s, 490 objects/s recovering 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:38.991 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:38.992 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-mon[40587]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:46:39.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.095 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:39.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:39.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:39.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:38 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:38 smithi067 ceph-mon[33346]: pgmap v844: 168 pgs: 1 active+undersized+remapped+backfilling, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 990 op/s; 418/144210 objects degraded (0.290%); 18262/144210 objects misplaced (12.663%); 25 MiB/s, 490 objects/s recovering 2024-03-26T21:46:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:38 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:38 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:39.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:38 smithi067 ceph-mon[33346]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T21:46:39.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:39.338 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:39.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:39.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:39.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:39.656 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:39.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:39.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:39.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:39.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:39.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:40.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:40.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:39 smithi055 ceph-mon[40587]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:46:40.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:39 smithi055 ceph-mon[40587]: 10.23 continuing backfill to osd.4 from (0'0,402'6415] 10:c5095edc:::benchmark_data_smithi055_163313_object48784:head to 402'6415 2024-03-26T21:46:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:39 smithi055 ceph-mon[29406]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:46:40.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:39 smithi055 ceph-mon[29406]: 10.23 continuing backfill to osd.4 from (0'0,402'6415] 10:c5095edc:::benchmark_data_smithi055_163313_object48784:head to 402'6415 2024-03-26T21:46:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:40.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.211 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:40.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:39 smithi067 ceph-mon[33346]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T21:46:40.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:39 smithi067 ceph-mon[33346]: 10.23 continuing backfill to osd.4 from (0'0,402'6415] 10:c5095edc:::benchmark_data_smithi055_163313_object48784:head to 402'6415 2024-03-26T21:46:40.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:40.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:40.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:40.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:40.567 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:40.594 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:40.696 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:40.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:40.822 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:40.848 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:40.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:40.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:41.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:40 smithi055 ceph-mon[40587]: pgmap v847: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 418/139239 objects degraded (0.300%); 15839/139239 objects misplaced (11.375%); 26 MiB/s, 614 objects/s recovering 2024-03-26T21:46:41.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:40 smithi055 ceph-mon[40587]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:46:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:40 smithi055 ceph-mon[29406]: pgmap v847: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 418/139239 objects degraded (0.300%); 15839/139239 objects misplaced (11.375%); 26 MiB/s, 614 objects/s recovering 2024-03-26T21:46:41.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:40 smithi055 ceph-mon[29406]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:46:41.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:41.106 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:40 smithi067 ceph-mon[33346]: pgmap v847: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 158 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 418/139239 objects degraded (0.300%); 15839/139239 objects misplaced (11.375%); 26 MiB/s, 614 objects/s recovering 2024-03-26T21:46:41.106 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:40 smithi067 ceph-mon[33346]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T21:46:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.132 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:41.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:41.233 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.267 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:41.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:41.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.396 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:41.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:41.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:41.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:41.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:41.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:41.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.775 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:41.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:41.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:41.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:41.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:42.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:42.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:42.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:42.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:42.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:41 smithi067 ceph-mon[33346]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:46:42.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:42.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:41 smithi055 ceph-mon[40587]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:46:42.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:41 smithi055 ceph-mon[29406]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T21:46:42.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:42.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:42.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:42.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:42.707 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:42.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:42.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:42.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:46:42.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T21:46:42.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:43.012 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:43.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:43.142 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:42 smithi067 ceph-mon[33346]: pgmap v850: 168 pgs: 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 160 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 418/137172 objects degraded (0.305%); 14543/137172 objects misplaced (10.602%); 27 MiB/s, 460 objects/s recovering 2024-03-26T21:46:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 418/137172 objects degraded (0.305%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:42 smithi067 ceph-mon[33346]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:46:43.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:43.279 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[29406]: pgmap v850: 168 pgs: 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 160 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 418/137172 objects degraded (0.305%); 14543/137172 objects misplaced (10.602%); 27 MiB/s, 460 objects/s recovering 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 418/137172 objects degraded (0.305%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[29406]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[40587]: pgmap v850: 168 pgs: 1 creating+remapped+peering, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 160 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 587 op/s; 418/137172 objects degraded (0.305%); 14543/137172 objects misplaced (10.602%); 27 MiB/s, 460 objects/s recovering 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 418/137172 objects degraded (0.305%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:42 smithi055 ceph-mon[40587]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T21:46:43.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:43.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:43.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:43.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:43.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:43.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:43.811 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:43.912 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:43.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:46:43.938 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:43.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:44.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:44.078 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:44.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:44.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:44.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:44.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:44.492 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:44.504 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:44.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:44.653 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:44.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:44.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:44.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:44.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:44 smithi067 ceph-mon[33346]: pgmap v852: 168 pgs: 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 161 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 831 op/s; 418/129231 objects degraded (0.323%); 14033/129231 objects misplaced (10.859%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:46:44.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:44 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:45.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:45.088 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:45.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:45.219 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:44 smithi055 ceph-mon[40587]: pgmap v852: 168 pgs: 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 161 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 831 op/s; 418/129231 objects degraded (0.323%); 14033/129231 objects misplaced (10.859%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:46:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:44 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:44 smithi055 ceph-mon[29406]: pgmap v852: 168 pgs: 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 161 active+clean; 2.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 831 op/s; 418/129231 objects degraded (0.323%); 14033/129231 objects misplaced (10.859%); 19 MiB/s, 298 objects/s recovering 2024-03-26T21:46:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:44 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:46:45.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:45.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:45.451 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:45.517 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:45.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.660 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:45.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:45.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:45.799 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:45.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:45.934 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:45.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:46.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:46.081 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:46.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:46.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:46.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:46.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:46.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:46.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:46.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:46.633 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:46.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:46.777 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:46.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:46.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:47.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:47.046 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:47.147 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: pgmap v853: 168 pgs: 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 162 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 418/126918 objects degraded (0.329%); 13094/126918 objects misplaced (10.317%); 15 MiB/s, 319 objects/s recovering 2024-03-26T21:46:47.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:47.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:47.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:47.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 418/126918 objects degraded (0.329%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:47.148 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:46 smithi067 ceph-mon[33346]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:46:47.148 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:47.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: pgmap v853: 168 pgs: 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 162 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 418/126918 objects degraded (0.329%); 13094/126918 objects misplaced (10.317%); 15 MiB/s, 319 objects/s recovering 2024-03-26T21:46:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:47.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 418/126918 objects degraded (0.329%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[29406]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: pgmap v853: 168 pgs: 5 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 162 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 685 op/s; 418/126918 objects degraded (0.329%); 13094/126918 objects misplaced (10.317%); 15 MiB/s, 319 objects/s recovering 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 418/126918 objects degraded (0.329%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:47.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:46 smithi055 ceph-mon[40587]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T21:46:47.408 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:47.448 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:47.549 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:47.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:47.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:47.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:47.855 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:47.888 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:47.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:47.997 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:47.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:48.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:48.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:48.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:48.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:48.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:48.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:46:48.260 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:48.362 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:48.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:48.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.511 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:48.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:48.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.636 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:48.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:48.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.761 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:48.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:48.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:48.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:48.988 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:49.026 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:49.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:49.216 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:48 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:49.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:48 smithi067 ceph-mon[33346]: pgmap v855: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.11k op/s; 222/116367 objects degraded (0.191%); 12023/116367 objects misplaced (10.332%); 17 MiB/s, 374 objects/s recovering 2024-03-26T21:46:49.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:48 smithi067 ceph-mon[33346]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:46:49.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:49.283 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:49.283 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[29406]: pgmap v855: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.11k op/s; 222/116367 objects degraded (0.191%); 12023/116367 objects misplaced (10.332%); 17 MiB/s, 374 objects/s recovering 2024-03-26T21:46:49.283 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[29406]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:46:49.284 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:49.284 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[40587]: pgmap v855: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 1.11k op/s; 222/116367 objects degraded (0.191%); 12023/116367 objects misplaced (10.332%); 17 MiB/s, 374 objects/s recovering 2024-03-26T21:46:49.284 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-mon[40587]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T21:46:49.284 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:49.314 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:49.321 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.14454.0:2868 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94208173348352 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.658276+0000", 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 343.66790515100001, 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0337873470000001, 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-03-26T21:46:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.14454", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2868 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658276+0000", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658276+0000", 2024-03-26T21:46:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658277+0000", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658283+0000", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8579999999999996e-06 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658292+0000", 2024-03-26T21:46:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5350000000000004e-06 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658299+0000", 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9569999999999996e-06 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658358+0000", 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8906000000000001e-05 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:46:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.658364+0000", 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7109999999999998e-06 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.664097+0000", 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057321769999999998 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.664107+0000", 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0543000000000001e-05 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.685532+0000", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.021424492 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.685933+0000", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040162200000000002 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.686899+0000", 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096622199999999998 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.687002+0000", 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102743 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.692063+0000", 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050613510000000004 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147846 9.7 9:e538dcb4:::benchmark_data_smithi055_135863_object17195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.668602+0000", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 343.65757955700002, 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1795881210000001, 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147846 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668602+0000", 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668602+0000", 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668602+0000", 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8400000000000005e-07 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668605+0000", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.852e-06 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668611+0000", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.976e-06 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.668617+0000", 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0120000000000002e-06 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.672769+0000", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041520369999999999 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.672775+0000", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0279999999999997e-06 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.673994+0000", 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.001219106 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.674009+0000", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4465e-05 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.837231+0000", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.16322221 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.837327+0000", 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6576999999999994e-05 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.837629+0000", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000301534 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.847896+0000", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010266498000000001 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.847950+0000", 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4015999999999999e-05 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.848007+0000", 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7639999999999997e-05 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.848074+0000", 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6552999999999998e-05 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.848190+0000", 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011613299999999999 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147852 9.7 9:e90e7024:::benchmark_data_smithi055_135863_object17201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.706357+0000", 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 343.619824687, 2024-03-26T21:46:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1667249239999999, 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147852 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706357+0000", 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706357+0000", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706358+0000", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5799999999999998e-07 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706362+0000", 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1529999999999999e-06 2024-03-26T21:46:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706372+0000", 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0258000000000001e-05 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706380+0000", 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6969999999999993e-06 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706421+0000", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1254000000000002e-05 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.706428+0000", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8360000000000003e-06 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.674066+0000", 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96763865500000001 2024-03-26T21:46:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.674267+0000", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020036300000000001 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.842139+0000", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1678724819999999 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.842227+0000", 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7519e-05 2024-03-26T21:46:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.842311+0000", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4406999999999993e-05 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.847903+0000", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055913839999999996 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.851557+0000", 2024-03-26T21:46:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003654189 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.872889+0000", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021332026 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.872950+0000", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1599000000000004e-05 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.873082+0000", 2024-03-26T21:46:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131144 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166760 9.a 9:5352ee92:::benchmark_data_smithi055_135863_object36109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.577507+0000", 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 324.74867378699997, 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.110810281, 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166760 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577507+0000", 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577507+0000", 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577509+0000", 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5990000000000001e-06 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577515+0000", 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5520000000000004e-06 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577531+0000", 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6209e-05 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577542+0000", 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577599+0000", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6957999999999999e-05 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577717+0000", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011851899999999999 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.577834+0000", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116929 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.580894+0000", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003059815 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.602252+0000", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021358644 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.602268+0000", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5489999999999999e-05 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.600988+0000", 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99872038699999999 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.601005+0000", 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6368999999999999e-05 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.688241+0000", 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0872364649999999 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.688318+0000", 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6637999999999994e-05 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166766 9.7 9:e7d331ac:::benchmark_data_smithi055_135863_object36115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.592319+0000", 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 324.73386214099997, 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1571443379999999, 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166766 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592319+0000", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592319+0000", 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592321+0000", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6789999999999999e-06 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592326+0000", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5140000000000001e-06 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592337+0000", 2024-03-26T21:46:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1105e-05 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.592349+0000", 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1430999999999999e-05 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616719+0000", 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024370269 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616728+0000", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9390000000000003e-06 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.737693+0000", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.120965121 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.737849+0000", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015626099999999999 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3,7", 2024-03-26T21:46:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.738022+0000", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172733 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.740604+0000", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025817150000000001 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.742756+0000", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021523620000000001 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.743202+0000", 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044526800000000002 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749264+0000", 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006061973 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749332+0000", 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8887999999999995e-05 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749463+0000", 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013108 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166776 9.a 9:51c39ff4:::benchmark_data_smithi055_135863_object36125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.616141+0000", 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 324.71004059099999, 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.078679395, 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166776 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616141+0000", 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616141+0000", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616141+0000", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3e-07 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616146+0000", 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5199999999999999e-06 2024-03-26T21:46:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616154+0000", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616161+0000", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3000000000000004e-06 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616241+0000", 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9913999999999994e-05 2024-03-26T21:46:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.616251+0000", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0009e-05 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.601043+0000", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98479143000000002 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.601053+0000", 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9580000000000002e-06 2024-03-26T21:46:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.688535+0000", 2024-03-26T21:46:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.087482689 2024-03-26T21:46:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.688686+0000", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015066299999999999 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.688832+0000", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014563600000000001 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694406+0000", 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055741590000000004 2024-03-26T21:46:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694426+0000", 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9949000000000002e-05 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694648+0000", 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000222253 2024-03-26T21:46:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694713+0000", 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4938999999999996e-05 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694820+0000", 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107093 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166785 9.a 9:5209f959:::benchmark_data_smithi055_135863_object36134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.630134+0000", 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 324.69604704400001, 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0721874900000001, 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166785 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630134+0000", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630134+0000", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630135+0000", 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.187e-06 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630141+0000", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7520000000000002e-06 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630151+0000", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0169e-05 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630162+0000", 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0329999999999999e-05 2024-03-26T21:46:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630222+0000", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0634000000000003e-05 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.630230+0000", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9000000000000006e-06 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.601081+0000", 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97085080700000004 2024-03-26T21:46:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.601091+0000", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4849999999999998e-06 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.691420+0000", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.090329449 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.691574+0000", 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015445500000000001 2024-03-26T21:46:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.691724+0000", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149302 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694601+0000", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028774299999999998 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694872+0000", 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027049300000000001 2024-03-26T21:46:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702099+0000", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007227051 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702207+0000", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108578 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702322+0000", 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114468 2024-03-26T21:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166792 9.a 9:5243b25f:::benchmark_data_smithi055_135863_object36141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.731933+0000", 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 323.59424817000001, 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97075458100000001, 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166792 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731933+0000", 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731933+0000", 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731935+0000", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.607e-06 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731941+0000", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3119999999999999e-06 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731952+0000", 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0845e-05 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731963+0000", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0665999999999999e-05 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.732018+0000", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5414000000000002e-05 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.732036+0000", 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8258e-05 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.694905+0000", 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96286830400000001 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.695064+0000", 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015904100000000001 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.695204+0000", 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014090399999999999 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.701773+0000", 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065685300000000004 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702382+0000", 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00060883400000000005 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702427+0000", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4880999999999999e-05 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702569+0000", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014218299999999999 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702688+0000", 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011880200000000001 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166795 9.a 9:56257fe8:::benchmark_data_smithi055_135863_object36144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.749712+0000", 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 323.57646949700001, 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.966939667, 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166795 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749712+0000", 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749712+0000", 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749713+0000", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.31e-06 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749719+0000", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3099999999999997e-06 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749730+0000", 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0421999999999999e-05 2024-03-26T21:46:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749738+0000", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6049999999999998e-06 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749808+0000", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9709999999999995e-05 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.749820+0000", 2024-03-26T21:46:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2002e-05 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.698471+0000", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94865115600000005 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.698638+0000", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016645700000000001 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.698810+0000", 2024-03-26T21:46:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017272300000000001 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702337+0000", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035269070000000001 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702738+0000", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000400426 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716487+0000", 2024-03-26T21:46:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013748692 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716558+0000", 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1749999999999996e-05 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716651+0000", 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3196999999999996e-05 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166796 9.a 9:5dc0cfc8:::benchmark_data_smithi055_135863_object36145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.753370+0000", 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 323.572811484, 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96350985099999997, 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166796 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753370+0000", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753370+0000", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753371+0000", 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.544e-06 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753378+0000", 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5660000000000003e-06 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753389+0000", 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0644e-05 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753397+0000", 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3699999999999995e-06 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753538+0000", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014097599999999999 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.753549+0000", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0908e-05 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702773+0000", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.949224654 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.702930+0000", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015677399999999999 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.703103+0000", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017256899999999999 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.709987+0000", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006884111 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.710007+0000", 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0567000000000001e-05 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716705+0000", 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066973309999999999 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716777+0000", 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2223000000000003e-05 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716880+0000", 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102614 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166810 9.a 9:5919c7d8:::benchmark_data_smithi055_135863_object36159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:25.774906+0000", 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 323.551275401, 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94220764800000001, 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166810 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774906+0000", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774906+0000", 2024-03-26T21:46:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774908+0000", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6249999999999999e-06 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774915+0000", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9999999999999999e-06 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774927+0000", 2024-03-26T21:46:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2741999999999999e-05 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.774937+0000", 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5850000000000006e-06 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.775006+0000", 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9504000000000004e-05 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.775018+0000", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1579e-05 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.706395+0000", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93137689999999995 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.706535+0000", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140593 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.706702+0000", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000166844 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.710015+0000", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003312742 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.712944+0000", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029292680000000001 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716936+0000", 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039913259999999999 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.716993+0000", 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7602999999999997e-05 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:26.717114+0000", 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120337 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:171699 9.a 9:56a3207d:::benchmark_data_smithi055_135863_object41048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:32.242296+0000", 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 317.08388508000002, 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.19018633800000001, 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 171699 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242296+0000", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242296+0000", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242298+0000", 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3820000000000001e-06 2024-03-26T21:46:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242304+0000", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1659999999999999e-06 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242314+0000", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.01e-05 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242323+0000", 2024-03-26T21:46:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7739999999999993e-06 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242388+0000", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5555000000000005e-05 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242523+0000", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013451199999999999 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.242644+0000", 2024-03-26T21:46:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120893 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.244804+0000", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002160361 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.244889+0000", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5437000000000002e-05 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.432052+0000", 2024-03-26T21:46:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.187162249 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.432279+0000", 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227765 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:32.432483+0000", 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020314399999999999 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:204417 9.7 9:ec63f8b0:::benchmark_data_smithi055_135863_object73766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:05.091298+0000", 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 284.23488290500001, 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.78498500199999999, 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 204417 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091298+0000", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091298+0000", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091300+0000", 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6810000000000001e-06 2024-03-26T21:46:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091306+0000", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.319e-06 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091321+0000", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4586999999999999e-05 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091330+0000", 2024-03-26T21:46:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1759999999999999e-06 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091391+0000", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0446999999999999e-05 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.091402+0000", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1547e-05 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.864916+0000", 2024-03-26T21:46:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.77351357700000001 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.865046+0000", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001303 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.865171+0000", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012504500000000001 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.870465+0000", 2024-03-26T21:46:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052934729999999999 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.870478+0000", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3541e-05 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.876088+0000", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056097839999999996 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.876179+0000", 2024-03-26T21:46:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0860999999999996e-05 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.876283+0000", 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104664 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:204438 9.7 9:ef00c0e6:::benchmark_data_smithi055_135863_object73787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:05.130362+0000", 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 284.19581933299997, 2024-03-26T21:46:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.75138349699999996, 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 204438 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130362+0000", 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130362+0000", 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130363+0000", 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1939999999999999e-06 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130369+0000", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3839999999999996e-06 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130379+0000", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0461e-05 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130385+0000", 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3239999999999999e-06 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130443+0000", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7553000000000003e-05 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.130453+0000", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0117e-05 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.867641+0000", 2024-03-26T21:46:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.73718755999999996 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.867747+0000", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106294 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.867862+0000", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115006 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.875870+0000", 2024-03-26T21:46:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080079890000000001 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881547+0000", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056772999999999997 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881590+0000", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2514e-05 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881645+0000", 2024-03-26T21:46:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5648e-05 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881745+0000", 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100153 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:204454 9.7 9:e5ed0afc:::benchmark_data_smithi055_135863_object73803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:05.165998+0000", 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 284.160183066, 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.72233331099999998, 2024-03-26T21:46:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 204454 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.165998+0000", 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.165998+0000", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166000+0000", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.595e-06 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166005+0000", 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5140000000000001e-06 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166018+0000", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2261e-05 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166031+0000", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400999999999999e-05 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166096+0000", 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4993000000000006e-05 2024-03-26T21:46:49.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.166106+0000", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5459999999999997e-06 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.870508+0000", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.70440228299999996 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.870627+0000", 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118725 2024-03-26T21:46:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.870700+0000", 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3910000000000002e-05 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.878415+0000", 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077148010000000003 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881782+0000", 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003366431 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.888154+0000", 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063725300000000004 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.888216+0000", 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1748999999999994e-05 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.888332+0000", 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115572 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:204464 9.7 9:ef5d0249:::benchmark_data_smithi055_135863_object73813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:05.206693+0000", 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 284.119488273, 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.68234990299999998, 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 204464 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206693+0000", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206693+0000", 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206694+0000", 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.449e-06 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206700+0000", 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3489999999999999e-06 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206710+0000", 2024-03-26T21:46:49.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0223e-05 2024-03-26T21:46:49.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:49.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206716+0000", 2024-03-26T21:46:49.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9599999999999997e-06 2024-03-26T21:46:49.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206762+0000", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6279000000000002e-05 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.206770+0000", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.267e-06 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.873123+0000", 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.66635327799999999 2024-03-26T21:46:49.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.873256+0000", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013325699999999999 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.873386+0000", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013035 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881509+0000", 2024-03-26T21:46:49.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081225929999999991 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.881818+0000", 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030910300000000003 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.888950+0000", 2024-03-26T21:46:49.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007132106 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.888995+0000", 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4864999999999998e-05 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:05.889043+0000", 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7824000000000002e-05 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52105 10.19 10:9f9f0187:::benchmark_data_smithi055_163313_object52104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337065+0000", 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.989116771, 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0288365690000001, 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52105 2024-03-26T21:46:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337065+0000", 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337065+0000", 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337072+0000", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4560000000000003e-06 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337117+0000", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4974e-05 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337139+0000", 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2234000000000001e-05 2024-03-26T21:46:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337151+0000", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2254e-05 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.454818+0000", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.117666438 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.454829+0000", 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0964e-05 2024-03-26T21:46:49.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360527+0000", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.90569817500000005 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360674+0000", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014731600000000001 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360839+0000", 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016456399999999999 2024-03-26T21:46:49.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.363086+0000", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022468829999999999 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.363327+0000", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000240957 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.365794+0000", 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024668580000000002 2024-03-26T21:46:49.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.365854+0000", 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0056999999999998e-05 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.365901+0000", 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7438999999999998e-05 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.445 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52113 10.9 10:921093db:::benchmark_data_smithi055_163313_object52112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.512704+0000", 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.813476987, 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.842950686, 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52113 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512704+0000", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512704+0000", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512714+0000", 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4979999999999995e-06 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512750+0000", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6338999999999999e-05 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512777+0000", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6948999999999999e-05 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512791+0000", 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4372000000000001e-05 2024-03-26T21:46:49.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512871+0000", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9840999999999997e-05 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.512886+0000", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4686e-05 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.346243+0000", 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.83335724600000005 2024-03-26T21:46:49.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.346413+0000", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016972800000000001 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,7", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.346601+0000", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018792200000000001 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.348715+0000", 2024-03-26T21:46:49.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021137550000000001 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.355519+0000", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068043050000000001 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.355557+0000", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8495000000000003e-05 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.355611+0000", 2024-03-26T21:46:49.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3185e-05 2024-03-26T21:46:49.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.355655+0000", 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4365e-05 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52116 10.6 10:67cd8b60:::benchmark_data_smithi055_163313_object52115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.516449+0000", 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.80973192499999, 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.85053739299999997, 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.454 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52116 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516449+0000", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516449+0000", 2024-03-26T21:46:49.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516451+0000", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5629999999999999e-06 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516494+0000", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2836000000000001e-05 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516507+0000", 2024-03-26T21:46:49.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3655999999999999e-05 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516516+0000", 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9730000000000003e-06 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516596+0000", 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9492000000000004e-05 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.516610+0000", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4552000000000001e-05 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.353724+0000", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.83711359399999996 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.353879+0000", 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154549 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,4", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.354043+0000", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000164387 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.359499+0000", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054564239999999996 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360125+0000", 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062575700000000001 2024-03-26T21:46:49.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.366906+0000", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006780782 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.366943+0000", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6949999999999997e-05 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.366987+0000", 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3878000000000002e-05 2024-03-26T21:46:49.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:53751 10.15 10.751a0415 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:06.358293+0000", 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 162.96788855200001, 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.457331724, 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:46:49.461 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 53751 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358293+0000", 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358293+0000", 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358294+0000", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3120000000000001e-06 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358349+0000", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5288000000000003e-05 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358360+0000", 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0175e-05 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.358367+0000", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7300000000000005e-06 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.364848+0000", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064812439999999997 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.364862+0000", 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3278000000000001e-05 2024-03-26T21:46:49.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.815614+0000", 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.45075204000000002 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:06.815624+0000", 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0657e-05 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:46:49.465 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:46:49.504 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:49.605 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:49.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:49.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:49.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:49.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:49.910 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:49.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:50.013 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:50.041 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:50.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:50.178 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:50.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:49 smithi067 ceph-mon[33346]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:46:50.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:50.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:49 smithi055 ceph-mon[29406]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:46:50.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:49 smithi055 ceph-mon[40587]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T21:46:50.318 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:50.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:50.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:46:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:50.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:50.716 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.758 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:50.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:50.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:50.874 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:50.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:50.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:51.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:51.042 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:51.064 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:50 smithi055 ceph-mon[40587]: pgmap v858: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 222/111015 objects degraded (0.200%); 11119/111015 objects misplaced (10.016%); 4.7 MiB/s, 184 objects/s recovering 2024-03-26T21:46:51.064 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:50 smithi055 ceph-mon[29406]: pgmap v858: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 222/111015 objects degraded (0.200%); 11119/111015 objects misplaced (10.016%); 4.7 MiB/s, 184 objects/s recovering 2024-03-26T21:46:51.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:51.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:51.184 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:50 smithi067 ceph-mon[33346]: pgmap v858: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_toofull, 162 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 222/111015 objects degraded (0.200%); 11119/111015 objects misplaced (10.016%); 4.7 MiB/s, 184 objects/s recovering 2024-03-26T21:46:51.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:51.318 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:51.420 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:51.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:51.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:51.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:51.695 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:51.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:51.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:51.888 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:51.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:51.902 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:46:51.902 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:46:51.907 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:51.909 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:46:51.960 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:46:51.961 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:46:51.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:52.008 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:46:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.028 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:52.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:52.054 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:46:52.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4234273797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2376100400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4234273797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2376100400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:52.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:51 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:52.098 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:46:52.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:52.140 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:46:52.146 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:46:52.147 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:46:52.172 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4234273797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:51 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:46:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2376100400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:46:52.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:51 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T21:46:52.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:52.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:52.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:52.556 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.601 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:52.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:52.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:52.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:52.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:52.867 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:46:52.867 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:46:52.881 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:52.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:52.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:53.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:53.120 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:53.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[40587]: pgmap v859: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 11119/108549 objects misplaced (10.243%); 4.7 MiB/s, 144 objects/s recovering 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[40587]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[29406]: pgmap v859: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 11119/108549 objects misplaced (10.243%); 4.7 MiB/s, 144 objects/s recovering 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:46:53.202 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:53.203 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:52 smithi055 ceph-mon[29406]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:46:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:52 smithi067 ceph-mon[33346]: pgmap v859: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 11119/108549 objects misplaced (10.243%); 4.7 MiB/s, 144 objects/s recovering 2024-03-26T21:46:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:52 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:46:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:52 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 222/111015 objects degraded (0.200%), 1 pg degraded, 1 pg undersized) 2024-03-26T21:46:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:52 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:46:53.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:52 smithi067 ceph-mon[33346]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T21:46:53.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:53.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:53.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:53.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:53.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:53.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:53.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:46:53.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.847 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:53.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:53.997 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:53.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:54.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:54.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:54.176 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:54.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:54.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.427 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:54.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:54.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:54.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:54.701 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:54.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.853 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:54.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:54.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:54.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:55.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:54 smithi055 ceph-mon[29406]: pgmap v861: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 10401/96816 objects misplaced (10.743%); 0 B/s, 36 objects/s recovering 2024-03-26T21:46:55.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:54 smithi055 ceph-mon[40587]: pgmap v861: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 10401/96816 objects misplaced (10.743%); 0 B/s, 36 objects/s recovering 2024-03-26T21:46:55.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:55.131 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:55.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:46:55.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:46:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:54 smithi067 ceph-mon[33346]: pgmap v861: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 10401/96816 objects misplaced (10.743%); 0 B/s, 36 objects/s recovering 2024-03-26T21:46:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:46:55.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:55.398 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:55.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:55.541 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:55.643 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:55.683 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:55.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:55.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:55.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:55.966 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:55.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:56.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:56.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:56.163 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:46:56.163 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:46:56.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:56.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:56.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.363 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:56.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:56.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:56.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:56.652 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:56.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.799 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:56.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:56.813 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:46:56.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:56.962 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:56.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:56.964 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:56 smithi055 ceph-mon[29406]: pgmap v862: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9020/93489 objects misplaced (9.648%); 0 B/s, 28 objects/s recovering 2024-03-26T21:46:56.964 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:56 smithi055 ceph-mon[29406]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:46:57.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:46:57.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:57.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:57.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:56 smithi067 ceph-mon[33346]: pgmap v862: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9020/93489 objects misplaced (9.648%); 0 B/s, 28 objects/s recovering 2024-03-26T21:46:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:56 smithi067 ceph-mon[33346]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:46:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.248 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:57.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:57.254 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:56 smithi055 ceph-mon[40587]: pgmap v862: 168 pgs: 5 active+remapped+backfill_toofull, 163 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9020/93489 objects misplaced (9.648%); 0 B/s, 28 objects/s recovering 2024-03-26T21:46:57.255 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:56 smithi055 ceph-mon[40587]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T21:46:57.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:57.405 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:46:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.406 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:57.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:57.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:46:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:57.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:46:57.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:57.689 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:57.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:57.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:46:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:46:57.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:46:57.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:57.977 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:57.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:58.005 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T21:46:58.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:58.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:58.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:46:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1371441671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.260 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:46:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1371441671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1371441671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:46:58.363 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:58.403 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:58.505 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:58.545 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:58.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.689 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:58.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:58.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.833 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:58.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:58.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:46:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:58.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:46:59.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.116 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:59.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:59.217 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:46:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:46:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:46:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: pgmap v864: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.28k op/s; 7994/80178 objects misplaced (9.970%); 808 KiB/s, 40 objects/s recovering 2024-03-26T21:46:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:46:59.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:46:58 smithi067 ceph-mon[33346]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:46:59.249 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:46:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:46:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: pgmap v864: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.28k op/s; 7994/80178 objects misplaced (9.970%); 808 KiB/s, 40 objects/s recovering 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[40587]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:46:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: pgmap v864: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.28k op/s; 7994/80178 objects misplaced (9.970%); 808 KiB/s, 40 objects/s recovering 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:46:59.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:46:58 smithi055 ceph-mon[29406]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T21:46:59.351 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:59.388 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:59.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:46:59.585 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:46:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:46:59.687 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:46:59.728 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:46:59.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:46:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:46:59.879 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:46:59.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:46:59.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:00.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:00.106 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:00.106 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:00.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:00.163 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:00.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:00.299 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:00.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:00.401 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:00.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T21:47:00.474 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:00.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.648 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:00.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:00.749 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:00.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:00.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:47:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:00.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:00.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:01.015 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:01.074 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:01.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:01.130 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:01 smithi055 ceph-mon[40587]: pgmap v866: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6116/73722 objects misplaced (8.296%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:47:01.130 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:01 smithi055 ceph-mon[29406]: pgmap v866: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6116/73722 objects misplaced (8.296%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:47:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:01.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:01.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:01 smithi067 ceph-mon[33346]: pgmap v866: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 6116/73722 objects misplaced (8.296%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:47:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:01.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:01.316 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:01.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.420 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:01.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:01.521 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:01.565 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:01.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:01.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:01.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:01.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:01.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:01.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:02.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:02.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:02.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:02.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:02.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:02.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:02.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:02.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:02.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:02.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:02.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:47:03.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:03.217 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: pgmap v867: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6116/71088 objects misplaced (8.603%); 3.3 MiB/s, 52 objects/s recovering 2024-03-26T21:47:03.217 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:47:03.217 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3720532198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: pgmap v867: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6116/71088 objects misplaced (8.603%); 3.3 MiB/s, 52 objects/s recovering 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3720532198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:03.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:03.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: pgmap v867: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 6116/71088 objects misplaced (8.603%); 3.3 MiB/s, 52 objects/s recovering 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3720532198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:03.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:03.400 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:03.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:03.543 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:03.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:03.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:03.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.821 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:03.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:03.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:03.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:04.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:04.078 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:04.078 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:04.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:04.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:03 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:03 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:04.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:03 smithi067 ceph-mon[33346]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:47:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:04.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:04.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[40587]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:47:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:03 smithi055 ceph-mon[29406]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T21:47:04.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:04.476 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.500 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:04.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:04.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:04.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:04.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:04.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:04.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:47:04.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:04.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:04.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:05.024 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: pgmap v870: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 3894/59097 objects misplaced (6.589%); 13 MiB/s, 200 objects/s recovering 2024-03-26T21:47:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:47:05.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 10.15 scrub starts 2024-03-26T21:47:05.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 10.15 scrub ok 2024-03-26T21:47:05.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:47:05.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:47:05.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: pgmap v870: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 3894/59097 objects misplaced (6.589%); 13 MiB/s, 200 objects/s recovering 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 10.15 scrub starts 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 10.15 scrub ok 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:47:05.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:05 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:47:05.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:05.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:05.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:05.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: pgmap v870: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 3894/59097 objects misplaced (6.589%); 13 MiB/s, 200 objects/s recovering 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 10.15 scrub starts 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 10.15 scrub ok 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:47:05.301 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:05 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:47:05.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:05.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:05.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.537 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:05.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:05.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.664 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:05.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:05.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:05.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:05.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:47:05.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:05.902 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:47:05.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:05.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:05.922 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:47:05.942 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:47:05.962 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:47:05.982 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:47:06.004 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:47:06.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:06.032 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:47:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:06.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:06.076 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:47:06.080 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:47:06.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:06.178 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:06.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.20 scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.23 scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.20 scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.23 scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.1e scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.1e scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.10 scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 10.10 scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:47:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2758226807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:06.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[40587]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.20 scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.23 scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.20 scrub ok 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.23 scrub ok 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.1e scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.1e scrub ok 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.10 scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 10.10 scrub ok 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:47:06.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:47:06.319 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2758226807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:06.319 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:06 smithi055 ceph-mon[29406]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:47:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:06.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:06.432 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.20 scrub starts 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.23 scrub starts 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.20 scrub ok 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.23 scrub ok 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.1e scrub starts 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.1e scrub ok 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.10 scrub starts 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 10.10 scrub ok 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:47:06.438 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:47:06.439 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:47:06.439 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:47:06.439 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2758226807' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:06.439 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:06 smithi067 ceph-mon[33346]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T21:47:06.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:06.519 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:06.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:06.617 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:06.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:06.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:06.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:06.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:06.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:07.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:07.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:07.087 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T21:47:07.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.184 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1f scrub starts 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1f scrub ok 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: pgmap v873: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 1252/56475 objects misplaced (2.217%); 27 MiB/s, 425 objects/s recovering 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1b scrub starts 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1 scrub starts 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1 scrub ok 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: 10.1b scrub ok 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1253067094' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-mon[33346]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:47:07.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1f scrub starts 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1f scrub ok 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: pgmap v873: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 1252/56475 objects misplaced (2.217%); 27 MiB/s, 425 objects/s recovering 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1b scrub starts 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1 scrub starts 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1 scrub ok 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: 10.1b scrub ok 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1253067094' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[40587]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1f scrub starts 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1f scrub ok 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: pgmap v873: 168 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 163 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 1252/56475 objects misplaced (2.217%); 27 MiB/s, 425 objects/s recovering 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1b scrub starts 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1 scrub starts 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1 scrub ok 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:47:07.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: 10.1b scrub ok 2024-03-26T21:47:07.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1253067094' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:47:07.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:07 smithi055 ceph-mon[29406]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:47:07.325 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:07.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.467 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:07.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.607 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:07.708 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:07.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:07.744 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:07.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:07.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:07.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:07.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:08.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:08.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:08.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:08.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:08.298 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:08.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:08.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.1e scrub starts 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.1c scrub starts 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.1e scrub ok 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.1c scrub ok 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:47:08.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:47:08.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:47:08.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.e scrub starts 2024-03-26T21:47:08.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:47:08.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:08 smithi067 ceph-mon[33346]: 10.e scrub ok 2024-03-26T21:47:08.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:08.537 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:47:08.538 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:47:08.538 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:47:08.538 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:47:08.538 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:47:08.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.1e scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.1c scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.1e scrub ok 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.1c scrub ok 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.e scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[40587]: 10.e scrub ok 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.1e scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.1c scrub starts 2024-03-26T21:47:08.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.1e scrub ok 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.1c scrub ok 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.e scrub starts 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-mon[29406]: 10.e scrub ok 2024-03-26T21:47:08.568 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:08.630 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T21:47:08.669 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:08.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:08.820 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:08.873 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:08.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:08.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.015 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:09.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:09.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:47:09.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:09.149 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:09.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:09.310 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:09.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:09.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 163 active+clean; 1009 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1242/49026 objects misplaced (2.533%) 2024-03-26T21:47:09.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:47:09.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 10.21 scrub starts 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 10.21 scrub ok 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 163 active+clean; 1009 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1242/49026 objects misplaced (2.533%) 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:47:09.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 10.21 scrub starts 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 10.21 scrub ok 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:09 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:47:09.317 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:09.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: pgmap v876: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 163 active+clean; 1009 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1242/49026 objects misplaced (2.533%) 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 10.21 scrub starts 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 10.21 scrub ok 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:47:09.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:09 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:47:09.520 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:09.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:09.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:09.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:09.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:09.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:09.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:09.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:09.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:10.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:10.117 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:10.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.243 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:10.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:10.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 10.b scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 10.b scrub ok 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 10.1d scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[40587]: 10.1d scrub ok 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 10.b scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 10.b scrub ok 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 10.1d scrub starts 2024-03-26T21:47:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:10 smithi055 ceph-mon[29406]: 10.1d scrub ok 2024-03-26T21:47:10.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:10.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 10.b scrub starts 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 10.b scrub ok 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 10.1d scrub starts 2024-03-26T21:47:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:10 smithi067 ceph-mon[33346]: 10.1d scrub ok 2024-03-26T21:47:10.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.527 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:10.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:10.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:10.665 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:10.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:10.798 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:10.842 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:10.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:10.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:11.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.064 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:11.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:11.134 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:47:11.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:11.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:11.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.331 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:11.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: pgmap v877: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 903 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 394/43896 objects misplaced (0.898%) 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 10.1a scrub starts 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 10.1a scrub ok 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/300213359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:47:11.393 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:47:11.394 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 10.17 scrub starts 2024-03-26T21:47:11.394 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:11 smithi067 ceph-mon[33346]: 10.17 scrub ok 2024-03-26T21:47:11.432 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:11.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: pgmap v877: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 903 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 394/43896 objects misplaced (0.898%) 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 10.1a scrub starts 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 10.1a scrub ok 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/300213359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 10.17 scrub starts 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[29406]: 10.17 scrub ok 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: pgmap v877: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 903 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 394/43896 objects misplaced (0.898%) 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 10.1a scrub starts 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 10.1a scrub ok 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/300213359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 10.17 scrub starts 2024-03-26T21:47:11.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:11 smithi055 ceph-mon[40587]: 10.17 scrub ok 2024-03-26T21:47:11.570 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:11.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:11.707 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.741 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:11.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:11.842 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:11.885 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:11.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:11.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:12.018 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:12.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:12.350 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:12.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:12.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 10.a scrub starts 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 10.a scrub ok 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:12.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:47:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:47:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:47:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:47:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:12 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:47:12.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 10.a scrub starts 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 10.a scrub ok 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:47:12.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 10.a scrub starts 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 10.a scrub ok 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:47:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:12 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:47:12.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:12.625 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:12.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:12.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:12.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:12.902 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:12.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:13.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:13.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:13.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:47:13.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.183 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:13.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:13.284 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: pgmap v879: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 841 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 394/40956 objects misplaced (0.962%) 2024-03-26T21:47:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 10.12 deep-scrub starts 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 2.2 deep-scrub starts 2024-03-26T21:47:13.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[40587]: 2.2 deep-scrub ok 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: pgmap v879: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 841 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 394/40956 objects misplaced (0.962%) 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 10.12 deep-scrub starts 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 2.2 deep-scrub starts 2024-03-26T21:47:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:13 smithi055 ceph-mon[29406]: 2.2 deep-scrub ok 2024-03-26T21:47:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.319 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:13.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:13.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: pgmap v879: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 166 active+clean; 841 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 394/40956 objects misplaced (0.962%) 2024-03-26T21:47:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 10.12 deep-scrub starts 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 2.2 deep-scrub starts 2024-03-26T21:47:13.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:13 smithi067 ceph-mon[33346]: 2.2 deep-scrub ok 2024-03-26T21:47:13.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:13.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:13.735 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:13.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:13.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:13.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:13.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:14.017 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:14.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:14.157 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:14.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.298 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:14.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:14.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:14.436 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 10.5 scrub starts 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 10.5 scrub ok 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 10.0 scrub starts 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 10.0 scrub ok 2024-03-26T21:47:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:47:14.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:14 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:47:14.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:47:14.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:47:14.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 10.5 scrub starts 2024-03-26T21:47:14.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 10.5 scrub ok 2024-03-26T21:47:14.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 10.0 scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 10.0 scrub ok 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 10.5 scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 10.5 scrub ok 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 10.0 scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 10.0 scrub ok 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:47:14.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:14 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:47:14.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:14.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:14.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:14.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:14.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:14.855 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:14.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:14.993 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:14.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:15.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:15.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:15.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:15.270 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: pgmap v880: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 580 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 394/28425 objects misplaced (1.386%) 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 10.12 deep-scrub ok 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 10.d scrub starts 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 10.d scrub ok 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:15.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: pgmap v880: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 580 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 394/28425 objects misplaced (1.386%) 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 10.12 deep-scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 10.d scrub starts 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 10.d scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:15 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:15.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.425 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:15.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:15.480 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: pgmap v880: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 580 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 394/28425 objects misplaced (1.386%) 2024-03-26T21:47:15.480 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 10.12 deep-scrub ok 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 10.d scrub starts 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 10.d scrub ok 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:15.481 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:15 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:15.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:15.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:15.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:15.700 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:15.801 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:15.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:15.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:15.991 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:15.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147839 9.c 9:30ff1895:::benchmark_data_smithi055_135863_object17188:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.665199+0000", 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 370.40605089600001, 2024-03-26T21:47:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1776680740000001, 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147839 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665199+0000", 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665199+0000", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665200+0000", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7100000000000011e-07 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665205+0000", 2024-03-26T21:47:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1080000000000001e-06 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665216+0000", 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0757000000000001e-05 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665225+0000", 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7339999999999997e-06 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665281+0000", 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6443999999999997e-05 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.665290+0000", 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4819999999999994e-06 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.668738+0000", 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0034479160000001 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.668752+0000", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4272e-05 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.841904+0000", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173152234 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.842867+0000", 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096315599999999995 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147867 9.c 9:308da875:::benchmark_data_smithi055_135863_object17216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.734002+0000", 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 370.33724766199998, 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1332662309999999, 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147867 2024-03-26T21:47:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734002+0000", 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734002+0000", 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734004+0000", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1710000000000001e-06 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734009+0000", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.733e-06 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734023+0000", 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3699999999999999e-05 2024-03-26T21:47:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734041+0000", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7425999999999999e-05 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734091+0000", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0859000000000002e-05 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.734099+0000", 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1940000000000001e-06 2024-03-26T21:47:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.668805+0000", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93470623799999997 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.668813+0000", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0320000000000003e-06 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.843204+0000", 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1743910399999999 2024-03-26T21:47:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.843416+0000", 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021170499999999999 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.843591+0000", 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000175532 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.853093+0000", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095023850000000003 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.856452+0000", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033590009999999999 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.867103+0000", 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010650046999999999 2024-03-26T21:47:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.867169+0000", 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6562000000000004e-05 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.867269+0000", 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9605999999999995e-05 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166765 9.c 9:3d9d5cb7:::benchmark_data_smithi055_135863_object36114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.585407+0000", 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 351.48584307099998, 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1205742599999999, 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:47:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166765 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585407+0000", 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585407+0000", 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585409+0000", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585415+0000", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8259999999999998e-06 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585426+0000", 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688999999999999e-05 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585435+0000", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4370000000000007e-06 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585492+0000", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7269999999999999e-05 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585636+0000", 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144275 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.585800+0000", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016343099999999999 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.591800+0000", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060004849999999998 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.594573+0000", 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027725390000000001 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.611876+0000", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017302946 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.611889+0000", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3190000000000001e-05 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.705892+0000", 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.094003375 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.705981+0000", 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9180999999999999e-05 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166782 9.c 9:339ee54f:::benchmark_data_smithi055_135863_object36131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.624827+0000", 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 351.44642362399998, 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0992929650000001, 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166782 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624827+0000", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624827+0000", 2024-03-26T21:47:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624828+0000", 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.485e-06 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624833+0000", 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0440000000000003e-06 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624845+0000", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1945e-05 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624862+0000", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6901999999999998e-05 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624903+0000", 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0945000000000002e-05 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.624909+0000", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9279999999999998e-06 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.706270+0000", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0813609360000001 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.706412+0000", 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014192799999999999 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,3,7", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.706889+0000", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047776900000000001 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.713168+0000", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062782319999999999 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.713223+0000", 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4907000000000001e-05 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.713305+0000", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2577000000000006e-05 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.723921+0000", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010615555 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.723995+0000", 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4480999999999994e-05 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.724119+0000", 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012433100000000001 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52097 10.11 10:8c81562f:::benchmark_data_smithi055_163313_object52096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.338216+0000", 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 191.733034366, 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.042768135, 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52097 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338216+0000", 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338216+0000", 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338217+0000", 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.559e-06 2024-03-26T21:47:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338258+0000", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0658000000000003e-05 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338268+0000", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0332e-05 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338277+0000", 2024-03-26T21:47:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4540000000000007e-06 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.475435+0000", 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13715838999999999 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.475445+0000", 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6090000000000007e-06 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.372462+0000", 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.89701760399999997 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.374471+0000", 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002008662 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.374607+0000", 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013571699999999999 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.376842+0000", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022355370000000001 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.377327+0000", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048479600000000001 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.380863+0000", 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035362589999999999 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.380909+0000", 2024-03-26T21:47:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5472000000000001e-05 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.380984+0000", 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5086000000000006e-05 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52101 10.11 10:8830a89b:::benchmark_data_smithi055_163313_object52100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.338288+0000", 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 191.73296185000001, 2024-03-26T21:47:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0448608159999999, 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52101 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338288+0000", 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338288+0000", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338289+0000", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9999999999999997e-07 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338325+0000", 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5898000000000001e-05 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338332+0000", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.401e-06 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.338339+0000", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4980000000000003e-06 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.475468+0000", 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13712903600000001 2024-03-26T21:47:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.475476+0000", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2779999999999993e-06 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.375684+0000", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.90020825299999996 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.375803+0000", 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011867299999999999 2024-03-26T21:47:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,7", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.375941+0000", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138116 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.377007+0000", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010655040000000001 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.380129+0000", 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031222720000000002 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.383058+0000", 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029288180000000001 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.383107+0000", 2024-03-26T21:47:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9645000000000003e-05 2024-03-26T21:47:16.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.383149+0000", 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1823999999999998e-05 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:138202 10.17 10:e98536ec:::benchmark_data_smithi055_163313_object34153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-03-26T21:47:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:02.972045+0000", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.099205068000003, 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.073232272, 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138202 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972045+0000", 2024-03-26T21:47:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972045+0000", 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972046+0000", 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2610000000000001e-06 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972054+0000", 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8369999999999997e-06 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972065+0000", 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0495e-05 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:02.972075+0000", 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0766e-05 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:03.129007+0000", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15693110599999999 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:03.130289+0000", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012823820000000001 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:03.253548+0000", 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12325935 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:03.256273+0000", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027246190000000002 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.001771+0000", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.74549824200000003 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.003914+0000", 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021428469999999998 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.008985+0000", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005071229 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.035358+0000", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026372909 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.036263+0000", 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090441800000000004 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.037345+0000", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001082003 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.039385+0000", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020408119999999999 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:04.045277+0000", 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005891996 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:141365 10.17 10:ec441b3b:::benchmark_data_smithi055_163313_object37316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:08.240714+0000", 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.830536166000002, 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.128234279, 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141365 2024-03-26T21:47:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240714+0000", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240714+0000", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240715+0000", 2024-03-26T21:47:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.279e-06 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240724+0000", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3049999999999993e-06 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240736+0000", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1984000000000001e-05 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.240743+0000", 2024-03-26T21:47:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-06 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.344251+0000", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10350778300000001 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.346119+0000", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018686009999999999 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.589299+0000", 2024-03-26T21:47:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.24317959 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.591772+0000", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024735629999999998 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.338008+0000", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.74623598000000002 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.339975+0000", 2024-03-26T21:47:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001966279 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.344147+0000", 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041723949999999997 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.360101+0000", 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015954330999999999 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.360813+0000", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00071128399999999996 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.361691+0000", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087887200000000005 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.362914+0000", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012222229999999999 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.368948+0000", 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060345909999999997 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:141683 10.17 10:eb467743:::benchmark_data_smithi055_163313_object37634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:08.644074+0000", 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.427175835, 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.966695941, 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141683 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644074+0000", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644074+0000", 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644076+0000", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2359999999999999e-06 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644080+0000", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9439999999999996e-06 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644093+0000", 2024-03-26T21:47:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2027e-05 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.644101+0000", 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.422e-06 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.727593+0000", 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.083492520000000001 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:47:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:08.729931+0000", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023374839999999999 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.538319+0000", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.80838855099999996 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.540486+0000", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021667610000000001 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.545358+0000", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872075 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.559754+0000", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014395876 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.560166+0000", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041180500000000001 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.603871+0000", 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043705486000000002 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.605379+0000", 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015079309999999999 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:09.610770+0000", 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053908230000000003 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:142340 10.17 10:ebbbf01f:::benchmark_data_smithi055_163313_object38291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:10.295263+0000", 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.775987337000004, 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96118480299999998, 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 142340 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295263+0000", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295263+0000", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295264+0000", 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.13e-06 2024-03-26T21:47:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295269+0000", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9899999999999997e-06 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295279+0000", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0404000000000001e-05 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.295287+0000", 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3549999999999999e-06 2024-03-26T21:47:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.385142+0000", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089855344000000004 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.386409+0000", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012665210000000001 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.474875+0000", 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088466562999999998 2024-03-26T21:47:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:10.477082+0000", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022066149999999999 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.190162+0000", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.71308068999999996 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.191925+0000", 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001762792 2024-03-26T21:47:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.195810+0000", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038851480000000002 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.215865+0000", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020054628000000001 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.247098+0000", 2024-03-26T21:47:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031232884999999998 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.249064+0000", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019656460000000001 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.250734+0000", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001670185 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.256448+0000", 2024-03-26T21:47:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057139069999999998 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:146091 10.17 10:ea965668:::benchmark_data_smithi055_163313_object42042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:16.269013+0000", 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.80223763, 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.001831838, 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 146091 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269013+0000", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269013+0000", 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269014+0000", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4309999999999999e-06 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269020+0000", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5829999999999997e-06 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269037+0000", 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7864e-05 2024-03-26T21:47:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.269046+0000", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5140000000000001e-06 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.278613+0000", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095671330000000002 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.279699+0000", 2024-03-26T21:47:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001085868 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.292732+0000", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013033437 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:16.295508+0000", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027753930000000001 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.230982+0000", 2024-03-26T21:47:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93547412399999996 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.232726+0000", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017446359999999999 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.236548+0000", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038214410000000001 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.252674+0000", 2024-03-26T21:47:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016126080000000001 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.263406+0000", 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010732006000000001 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.264273+0000", 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086663400000000002 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.265860+0000", 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015872900000000001 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:17.270844+0000", 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049844039999999996 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:161493 10.21 10:8704c0c7:::benchmark_data_smithi055_163313_object57444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:36.834100+0000", 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.237150358999997, 2024-03-26T21:47:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.004484288, 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 161493 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834100+0000", 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834100+0000", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834101+0000", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.666e-06 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834105+0000", 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9920000000000001e-06 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834115+0000", 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6979999999999993e-06 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834121+0000", 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0299999999999999e-06 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834167+0000", 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5469999999999997e-05 2024-03-26T21:47:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.834172+0000", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3689999999999998e-06 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.834609+0000", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0004370680000001 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.834703+0000", 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3708000000000005e-05 2024-03-26T21:47:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,2", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.834903+0000", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000200535 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838381+0000", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003477295 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838438+0000", 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7099000000000003e-05 2024-03-26T21:47:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838480+0000", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2234999999999998e-05 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838538+0000", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7966e-05 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838584+0000", 2024-03-26T21:47:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6156999999999999e-05 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:161527 10.21 10:8738826f:::benchmark_data_smithi055_163313_object57478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:36.861572+0000", 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.209678042999997, 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97988217499999997, 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 161527 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861572+0000", 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861572+0000", 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861574+0000", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4440000000000001e-06 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861577+0000", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8920000000000002e-06 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861586+0000", 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3280000000000006e-06 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861592+0000", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6209999999999998e-06 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861681+0000", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9030999999999995e-05 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:36.861689+0000", 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.515e-06 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.836234+0000", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97454528500000004 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.836334+0000", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9293e-05 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0,2", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.836543+0000", 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020917 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838690+0000", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021476849999999999 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.838731+0000", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1004000000000002e-05 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.841338+0000", 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026065490000000001 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.841396+0000", 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8127000000000001e-05 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:37.841454+0000", 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8230999999999999e-05 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:164278 10.1d 10:bcf306cc:::benchmark_data_smithi055_163313_object60229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-03-26T21:47:16.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:39.859223+0000", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.212027114000001, 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829193720000001, 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 164278 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859223+0000", 2024-03-26T21:47:16.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859223+0000", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859224+0000", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2839999999999999e-06 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859229+0000", 2024-03-26T21:47:16.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5639999999999998e-06 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859238+0000", 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0920000000000004e-06 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.859245+0000", 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7739999999999999e-06 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.861250+0000", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020051779999999998 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.861264+0000", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3920999999999999e-05 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.937656+0000", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.076392008 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.937783+0000", 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012751 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.938055+0000", 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027179599999999998 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.940562+0000", 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025066889999999999 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.941966+0000", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014041360000000001 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942080+0000", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113588 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942114+0000", 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4112999999999997e-05 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942142+0000", 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8719000000000001e-05 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:164282 10.1d 10:b92069f9:::benchmark_data_smithi055_163313_object60233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:39.862360+0000", 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.208889882999998, 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.079931389, 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 164282 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862360+0000", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862360+0000", 2024-03-26T21:47:16.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862362+0000", 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0549999999999998e-06 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862367+0000", 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.481e-06 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862377+0000", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9890000000000004e-06 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862385+0000", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7619999999999995e-06 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862950+0000", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056530399999999998 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.862958+0000", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6459999999999999e-06 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.939499+0000", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.076540386 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.939608+0000", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108813 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:47:16.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.939774+0000", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016661599999999999 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942197+0000", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024231140000000001 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942213+0000", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5316e-05 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942241+0000", 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8234000000000001e-05 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942267+0000", 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6271e-05 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942292+0000", 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4402000000000001e-05 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:164370 10.1d 10:bc112582:::benchmark_data_smithi055_163313_object60321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:39.925895+0000", 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.145354722999997, 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.016513794, 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 164370 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.183 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925895+0000", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925895+0000", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925897+0000", 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2270000000000001e-06 2024-03-26T21:47:16.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925901+0000", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3669999999999999e-06 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925911+0000", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0125999999999999e-05 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925919+0000", 2024-03-26T21:47:16.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8769999999999993e-06 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925974+0000", 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4673999999999999e-05 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:39.925982+0000", 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6950000000000006e-06 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.940604+0000", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.014621776 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.940716+0000", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011224 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.940924+0000", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020793999999999999 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942219+0000", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294545 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942337+0000", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118175 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942363+0000", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5556999999999999e-05 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942387+0000", 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3907999999999999e-05 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942409+0000", 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2687000000000001e-05 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:164403 10.1d 10:bc6f8b71:::benchmark_data_smithi055_163313_object60354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:40.000204+0000", 2024-03-26T21:47:16.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.071046600999999, 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94689178699999998, 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 164403 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000204+0000", 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000204+0000", 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000205+0000", 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5519999999999999e-06 2024-03-26T21:47:16.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000210+0000", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9579999999999998e-06 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000349+0000", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139288 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000360+0000", 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0258000000000001e-05 2024-03-26T21:47:16.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000416+0000", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6229e-05 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.000425+0000", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6290000000000005e-06 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942446+0000", 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94202044699999998 2024-03-26T21:47:16.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942561+0000", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114702 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.942741+0000", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018068399999999999 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.946810+0000", 2024-03-26T21:47:16.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040688549999999997 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.946916+0000", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010604599999999999 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.946982+0000", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6286999999999997e-05 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.947052+0000", 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9825000000000003e-05 2024-03-26T21:47:16.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:40.947095+0000", 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3027e-05 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:172036 10.17 10:ee7e9286:::benchmark_data_smithi055_163313_object67987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:47.954506+0000", 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.116743909, 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97579937800000005, 2024-03-26T21:47:16.196 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172036 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954506+0000", 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954506+0000", 2024-03-26T21:47:16.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954507+0000", 2024-03-26T21:47:16.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1659999999999999e-06 2024-03-26T21:47:16.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954512+0000", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8409999999999999e-06 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954523+0000", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0613e-05 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954532+0000", 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5740000000000002e-06 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954742+0000", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020955799999999999 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.954754+0000", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.24e-05 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.925464+0000", 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97070926099999999 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.925555+0000", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1221e-05 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.925798+0000", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024271399999999999 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930170+0000", 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043727210000000004 2024-03-26T21:47:16.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930179+0000", 2024-03-26T21:47:16.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0610000000000002e-06 2024-03-26T21:47:16.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930215+0000", 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5250000000000003e-05 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930262+0000", 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7341e-05 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930306+0000", 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3656999999999999e-05 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:172082 10.17 10:ece21c77:::benchmark_data_smithi055_163313_object68033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:47.983882+0000", 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.087367988, 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94669545099999997, 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.205 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172082 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983882+0000", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983882+0000", 2024-03-26T21:47:16.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983883+0000", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.018e-06 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983887+0000", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1629999999999998e-06 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983896+0000", 2024-03-26T21:47:16.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9469999999999992e-06 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983902+0000", 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200000000000001e-06 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983965+0000", 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3134999999999998e-05 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.983975+0000", 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8260000000000005e-06 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.927635+0000", 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94365940000000004 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.927727+0000", 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2212000000000004e-05 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.927927+0000", 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020026199999999999 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930449+0000", 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025224380000000001 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930497+0000", 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7750000000000002e-05 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930511+0000", 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3969e-05 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930547+0000", 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6100999999999999e-05 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930578+0000", 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0309999999999999e-05 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:172099 10.17 10:ecb5a534:::benchmark_data_smithi055_163313_object68050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:47.996134+0000", 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.075116252000001, 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93456218000000002, 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:16.213 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172099 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996134+0000", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996134+0000", 2024-03-26T21:47:16.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996136+0000", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.658e-06 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996142+0000", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2179999999999996e-06 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996162+0000", 2024-03-26T21:47:16.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-05 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996171+0000", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1160000000000005e-06 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996236+0000", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5642000000000006e-05 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:47.996247+0000", 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0212e-05 2024-03-26T21:47:16.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.928787+0000", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93254067299999999 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.928904+0000", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116327 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,6", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.929133+0000", 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022981799999999999 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930585+0000", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001451349 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930618+0000", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3314e-05 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930651+0000", 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2493000000000002e-05 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930676+0000", 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5661999999999999e-05 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:48.930696+0000", 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9848e-05 2024-03-26T21:47:16.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:47:16.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:47:16.220 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:47:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:16.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:16.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:47:16.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:16.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 10.16 scrub starts 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 10.16 scrub ok 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 4.1d deep-scrub starts 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 4.1d deep-scrub ok 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 6.1 scrub starts 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 6.1 scrub ok 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:47:16.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 10.16 scrub starts 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 10.16 scrub ok 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 4.1d deep-scrub starts 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 4.1d deep-scrub ok 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 6.1 scrub starts 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 6.1 scrub ok 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:47:16.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:16 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:47:16.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 10.16 scrub starts 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 10.16 scrub ok 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 4.1d deep-scrub starts 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 4.1d deep-scrub ok 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 6.1 scrub starts 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 6.1 scrub ok 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:47:16.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:16 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:47:16.508 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:16.610 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:16.665 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:16.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:16.815 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:16.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:16.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:17.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:17.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:17.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:17.235 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:17.316 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:17.335 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:47:17.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:17.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: pgmap v882: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 536 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 232/26283 objects misplaced (0.883%) 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 10.f scrub starts 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 10.f scrub ok 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[29406]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: pgmap v882: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 536 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 232/26283 objects misplaced (0.883%) 2024-03-26T21:47:17.384 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 10.f scrub starts 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 10.f scrub ok 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:47:17.385 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:17 smithi055 ceph-mon[40587]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:47:17.385 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:47:17.435 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:47:17.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:17.480 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: pgmap v882: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 536 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 232/26283 objects misplaced (0.883%) 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 10.f scrub starts 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 10.f scrub ok 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:47:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-mon[33346]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T21:47:17.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:17.520 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:47:17.559 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:47:17.609 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:47:17.612 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:17.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:47:17.650 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:47:17.658 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:17.662 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:47:17.662 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:47:17.662 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:47:17.663 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:47:17.663 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:47:17.663 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:47:17.663 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:47:17.663 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:47:17.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:17.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:17.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:17.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:18.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:18.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:18.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:18.220 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T21:47:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:18.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:18.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:18.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:18.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3087582928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:18.488 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 3.1c deep-scrub starts 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 3.1c deep-scrub ok 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:47:18.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:18 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:47:18.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:18.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3087582928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 3.1c deep-scrub starts 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 3.1c deep-scrub ok 2024-03-26T21:47:18.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3087582928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 3.1c deep-scrub starts 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 3.1c deep-scrub ok 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:47:18.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:47:18.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:18.663 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:18.765 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:18.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:18.815 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Removed 104046 objects 2024-03-26T21:47:18.815 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Clean up completed and total clean up time :126.541 2024-03-26T21:47:18.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:18.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:19.028 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:19.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:19.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:19.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.181 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:19.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:19.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:19.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:19.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:19.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:19.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: pgmap v884: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 278 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 232/13914 objects misplaced (1.667%) 2024-03-26T21:47:19.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:47:19.472 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: pgmap v884: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 278 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 232/13914 objects misplaced (1.667%) 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:47:19.473 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:19 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: pgmap v884: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 278 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 624 B/s rd, 0 op/s; 232/13914 objects misplaced (1.667%) 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/548181139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:47:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:19 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:47:19.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:19.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:19.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.694 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:19.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:19.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:19.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:19.856 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:19.856 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:19.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:19.947 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:19.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:20.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.074 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:20.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:20.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:20.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:20.202 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T21:47:20.302 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:47:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:47:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:47:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:47:20.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 6.5 deep-scrub starts 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[40587]: 6.5 deep-scrub ok 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 6.5 deep-scrub starts 2024-03-26T21:47:20.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:20 smithi055 ceph-mon[29406]: 6.5 deep-scrub ok 2024-03-26T21:47:20.326 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:20.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:20.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:20.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:20.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:20.639 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-26T21:47:20.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-26T21:47:20.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 6.5 deep-scrub starts 2024-03-26T21:47:20.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:20 smithi067 ceph-mon[33346]: 6.5 deep-scrub ok 2024-03-26T21:47:20.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:20.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:20.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:20.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:20.975 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.002 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:21.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.128 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:21.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:21.356 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:21.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:21 smithi067 ceph-mon[33346]: pgmap v886: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 120 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6/6306 objects misplaced (0.095%) 2024-03-26T21:47:21.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:21 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:47:21.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:21 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:47:21.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:21 smithi067 ceph-mon[33346]: 5.18 scrub starts 2024-03-26T21:47:21.436 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:21 smithi067 ceph-mon[33346]: 5.18 scrub ok 2024-03-26T21:47:21.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:47:21.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.504 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:21.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[40587]: pgmap v886: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 120 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6/6306 objects misplaced (0.095%) 2024-03-26T21:47:21.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:47:21.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[40587]: 5.18 scrub starts 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[40587]: 5.18 scrub ok 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[29406]: pgmap v886: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 120 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6/6306 objects misplaced (0.095%) 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[29406]: 5.18 scrub starts 2024-03-26T21:47:21.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:21 smithi055 ceph-mon[29406]: 5.18 scrub ok 2024-03-26T21:47:21.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:21.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:21.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.756 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:21.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:21.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:21.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:21.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:21.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:22.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:22.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:22.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:22.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:22.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:22.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:22.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:22.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:22.512 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:22.514 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:22.518 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:22.614 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:22.638 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:22.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:47:22.740 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.764 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:22.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:22.865 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:22.891 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:22.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:22.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:23.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[29406]: pgmap v887: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 87 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 787 B/s rd, 0 op/s; 6/4764 objects misplaced (0.126%) 2024-03-26T21:47:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:47:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:47:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[29406]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:47:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[40587]: pgmap v887: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 87 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 787 B/s rd, 0 op/s; 6/4764 objects misplaced (0.126%) 2024-03-26T21:47:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:47:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:47:23.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:22 smithi055 ceph-mon[40587]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:47:23.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:23.161 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:22 smithi067 ceph-mon[33346]: pgmap v887: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 87 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 787 B/s rd, 0 op/s; 6/4764 objects misplaced (0.126%) 2024-03-26T21:47:23.162 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:22 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:47:23.162 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:22 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:47:23.162 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:22 smithi067 ceph-mon[33346]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T21:47:23.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:23.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:23.370 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.395 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:23.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:23.397 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:23.397 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:23.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.520 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:23.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:23.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:23.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:23.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:23.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:23.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:23.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:23.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:23.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:24.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 4.13 deep-scrub starts 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 4.13 deep-scrub ok 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:47:24.144 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:23 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:47:24.151 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:24.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:47:24.212 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 4.13 deep-scrub starts 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 4.13 deep-scrub ok 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 4.13 deep-scrub starts 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 4.13 deep-scrub ok 2024-03-26T21:47:24.213 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:47:24.214 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:47:24.214 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:47:24.214 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:47:24.214 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:47:24.214 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:23 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:47:24.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.277 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:24.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:24.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:47:24.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:24.529 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:24.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:24.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:24.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:24.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:24.883 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:24.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:24.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:25.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.035 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:25.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:25.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:47:25.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 10.6 scrub starts 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 10.6 scrub ok 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[29406]: pgmap v889: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:47:25.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:47:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 10.6 scrub starts 2024-03-26T21:47:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 10.6 scrub ok 2024-03-26T21:47:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:47:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:47:25.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:24 smithi055 ceph-mon[40587]: pgmap v889: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:25.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:25.159 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 10.6 scrub starts 2024-03-26T21:47:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 10.6 scrub ok 2024-03-26T21:47:25.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:47:25.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:47:25.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:24 smithi067 ceph-mon[33346]: pgmap v889: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:25.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:25.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:25.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:25.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:25.512 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:25.536 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:25.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:25.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:25.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:25.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:25.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:25.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:25.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:25.952 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:47:25.971 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/711120106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:25.972 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:25 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:26.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:26.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:26.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.165 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:26.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:26.240 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:26.240 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/711120106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:26.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/711120106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:26.265 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:25 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:47:26.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.290 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:26.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:26.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:26.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:26.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.541 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:26.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:26.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T21:47:26.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:26.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:26.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:26.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:26.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:26.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:47:27.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:27.043 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:27.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:27.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.170 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:27.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:27.221 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: pgmap v891: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 436 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 10.4 scrub starts 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: 10.4 scrub ok 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[40587]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:47:27.222 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: pgmap v891: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 436 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 10.4 scrub starts 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: 10.4 scrub ok 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:47:27.223 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:26 smithi055 ceph-mon[29406]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: pgmap v891: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 436 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 10.4 scrub starts 2024-03-26T21:47:27.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: 10.4 scrub ok 2024-03-26T21:47:27.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:47:27.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:26 smithi067 ceph-mon[33346]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T21:47:27.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:27.296 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:27.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:27.424 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:27.525 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:27.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:27.556 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:27.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:27.682 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:27.779 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:27.779 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:27.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:27.811 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:27.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:27.909 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_3' removed 2024-03-26T21:47:27.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:27.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:28.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:28.089 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:28.191 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:28.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/269542895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:47:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:27 smithi067 ceph-mon[33346]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/269542895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:47:28.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[40587]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/269542895' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:47:28.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:27 smithi055 ceph-mon[29406]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T21:47:28.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:28.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:28.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.467 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:28.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:28.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:28.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:28.693 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:28.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:28.779 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T21:47:28.779 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T21:47:28.780 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-26T21:47:28.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-26T21:47:28.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:28.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:28.909 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:47:28.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:28.972 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:28.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 4.e deep-scrub starts 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 4.e deep-scrub ok 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: pgmap v893: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 231 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 10.22 deep-scrub starts 2024-03-26T21:47:29.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 10.22 deep-scrub ok 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 4.e deep-scrub starts 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 4.e deep-scrub ok 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: pgmap v893: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 231 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 10.22 deep-scrub starts 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 10.22 deep-scrub ok 2024-03-26T21:47:29.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:47:29.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:47:29.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:28 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:47:29.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.099 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:29.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:29.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:29.228 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 4.e deep-scrub starts 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 4.e deep-scrub ok 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: pgmap v893: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 774 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 231 op/s; 6/603 objects misplaced (0.995%) 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 10.22 deep-scrub starts 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 10.22 deep-scrub ok 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:47:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:47:29.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:28 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:47:29.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.356 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:29.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:29.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:29.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:29.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:29.611 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:29.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:29.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:29.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:29.864 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:29.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:29.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:30.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:30.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:30.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:47:30.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:29 smithi067 ceph-mon[33346]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:47:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:30.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:47:30.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[29406]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:47:30.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:30.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4132126215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:47:30.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:29 smithi055 ceph-mon[40587]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T21:47:30.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:30.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:30.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:30.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.637 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:30.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:30.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:30.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:30.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:31.025 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:31.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:31.155 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:31.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:30 smithi067 ceph-mon[33346]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:31.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:30 smithi067 ceph-mon[33346]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:47:31.256 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:31.280 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:31.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:30 smithi055 ceph-mon[40587]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:31.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:30 smithi055 ceph-mon[40587]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:47:31.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:30 smithi055 ceph-mon[29406]: pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:31.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:30 smithi055 ceph-mon[29406]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:47:31.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:31.406 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:31.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:31.536 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:31.638 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:31.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:31.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:31.789 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:31.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:31.916 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:31.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:32.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:32.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.168 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:32.269 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:32.294 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:32.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.421 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:32.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.546 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:32.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:32.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:32.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:32.801 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:32.858 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:32.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:32.928 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:32.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:33.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:33.053 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[40587]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[29406]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:33.132 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:33.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:33.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:33.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:32 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:33.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:33.179 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:32 smithi067 ceph-mon[33346]: pgmap v897: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:32 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:32 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:32 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:32 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:33.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:33.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:33.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.431 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:33.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:33.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:33.560 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:33.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:33.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:33.817 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:33.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:33.943 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:33.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:34.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[40587]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[29406]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:34.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:33 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:34.069 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:34.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:33 smithi067 ceph-mon[33346]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T21:47:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:33 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:34.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:33 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:34.301 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:34.325 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:34.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:34.450 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:34.551 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:34.575 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:34.677 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:34.699 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:47:34.699 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:47:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:34.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:34.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:34.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:47:34.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:34.960 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:34.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:34.967 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T21:47:35.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:35.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.219 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:35.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:34 smithi067 ceph-mon[33346]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:47:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:34 smithi067 ceph-mon[33346]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:47:35.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:34 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:47:35.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:35 smithi055 ceph-mon[40587]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:47:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:35 smithi055 ceph-mon[40587]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:47:35.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:35 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:47:35.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:34 smithi055 ceph-mon[29406]: pgmap v899: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:47:35.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:34 smithi055 ceph-mon[29406]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T21:47:35.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:34 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:47:35.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.345 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:35.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:35.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:35.470 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:35.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:35.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:35.702 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:35.727 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:35.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:35.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:35.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:35.984 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:35.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:47:36.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:36 smithi055 ceph-mon[40587]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:47:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:47:36.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:36 smithi055 ceph-mon[29406]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:47:36.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:36.215 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.239 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:36.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:36.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:36.344 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T21:47:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.365 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:36.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:36.467 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2772144346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:47:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:36 smithi067 ceph-mon[33346]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T21:47:36.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:36.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:36.616 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:36.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:36.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:36.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:36.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:36.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:36.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:37.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:37.041 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:47:37.061 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:47:37.081 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:47:37.101 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:47:37.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:37.121 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:47:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:37.141 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:47:37.162 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:47:37.167 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:37 smithi055 ceph-mon[29406]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:37.167 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:37 smithi055 ceph-mon[29406]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:47:37.168 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:37 smithi055 ceph-mon[40587]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:37.168 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:37 smithi055 ceph-mon[40587]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:47:37.182 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:47:37.184 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:47:37.185 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:47:37.185 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:47:37.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.263 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:37.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:37.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:37 smithi067 ceph-mon[33346]: pgmap v902: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:37 smithi067 ceph-mon[33346]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T21:47:37.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:37.519 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:37.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:37.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:37.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:37.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:37.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:37.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:37.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:37.935 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:47:37.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:37.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.029 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:38.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:38.031 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_4' created 2024-03-26T21:47:38.130 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:38.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:38.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:38.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/639274454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:38.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T21:47:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/639274454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:38.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T21:47:38.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:38.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/639274454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T21:47:38.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.540 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:38.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:38.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:38.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:38.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-26T21:47:38.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:38.802 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:38.803 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:38.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:38.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:38.936 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:38.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:39.037 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.066 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:39.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:39.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:39.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:39.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[29406]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T21:47:39.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[29406]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:47:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[40587]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T21:47:39.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:39 smithi055 ceph-mon[40587]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:47:39.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:39.320 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:39.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:39.450 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:39.453 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:39.453 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:39 smithi067 ceph-mon[33346]: pgmap v904: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:39.453 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:39 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/180261006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T21:47:39.453 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:39 smithi067 ceph-mon[33346]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T21:47:39.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.582 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:39.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:39.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:39.708 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:39.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.836 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:39.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:39.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:39.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:39.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:40.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:40.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:40.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.217 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:40.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:40.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:40 smithi055 ceph-mon[40587]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:47:40.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:40 smithi055 ceph-mon[40587]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:47:40.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:40 smithi055 ceph-mon[29406]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:47:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:40 smithi055 ceph-mon[29406]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:47:40.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.344 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:40.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:40.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:40.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:40.723 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:40 smithi067 ceph-mon[33346]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T21:47:40.723 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:40 smithi067 ceph-mon[33346]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T21:47:40.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:40.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:40.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:40.974 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:40.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:41.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:41.088 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:41.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:41.124 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:41.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.224 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:41.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:41.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:41 smithi055 ceph-mon[29406]: pgmap v907: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:41.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:41 smithi055 ceph-mon[29406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:47:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:41 smithi055 ceph-mon[40587]: pgmap v907: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:41 smithi055 ceph-mon[40587]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:47:41.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:41.352 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:47:41.450 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:41.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:41 smithi067 ceph-mon[33346]: pgmap v907: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:41 smithi067 ceph-mon[33346]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T21:47:41.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:41.604 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:41.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.731 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:41.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:41.832 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.859 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:41.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:41.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:41.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:42.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:42.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:42.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:42.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:42.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:42.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:42.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:42.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.865 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:42.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:42.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:43.072 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:47:43.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:43.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:43.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:43.224 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[40587]: pgmap v909: 148 pgs: 4 creating+peering, 11 unknown, 133 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[40587]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1241698832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[29406]: pgmap v909: 148 pgs: 4 creating+peering, 11 unknown, 133 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[29406]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1241698832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.225 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:42 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:42 smithi067 ceph-mon[33346]: pgmap v909: 148 pgs: 4 creating+peering, 11 unknown, 133 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:47:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:42 smithi067 ceph-mon[33346]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T21:47:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:42 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1241698832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:42 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:47:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:43.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:43.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:43.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:43.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:43.597 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.624 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:43.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:43.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:43.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:43.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:43.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:43.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:43.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:44.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:44.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:44.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:44.060 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[40587]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:47:44.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:44.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:44.061 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:43 smithi055 ceph-mon[29406]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:47:44.105 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.129 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:44.230 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:43 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:43 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:47:44.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:43 smithi067 ceph-mon[33346]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T21:47:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.254 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:44.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:44.406 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:47:44.407 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:47:44.407 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:47:44.407 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:47:44.407 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:47:44.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:44.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.638 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:44.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:44.866 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:44.891 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:44.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:44.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:45.020 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:45.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:45.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:47:45.150 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:45.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.280 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:45.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:45.319 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:45 smithi055 ceph-mon[29406]: pgmap v912: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:45.319 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:45 smithi055 ceph-mon[40587]: pgmap v912: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:45.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:45.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:45 smithi067 ceph-mon[33346]: pgmap v912: 148 pgs: 1 creating+peering, 147 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:47:45.510 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:45.534 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:45.636 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.661 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:45.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:45.762 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.786 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:45.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:45.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:45.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:46.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:46.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:46.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:46.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:46.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:46.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:46.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:46.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.561 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:46.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:46.662 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:46.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:46.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:46.913 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:46.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:46.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:47.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.065 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:47.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:47.166 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:47.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:47.292 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:47.316 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:47.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[40587]: pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:47.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:47.409 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[40587]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:47:47.409 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[29406]: pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:47.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:47.410 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:47 smithi055 ceph-mon[29406]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:47:47.419 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:47.449 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:47.464 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:47.466 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:47 smithi067 ceph-mon[33346]: pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:47 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:47:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:47 smithi067 ceph-mon[33346]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T21:47:47.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:47.576 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:47.678 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:47.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:47.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.827 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:47.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:47.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:47.953 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:47.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:47.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:48.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:48.082 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T21:47:48.083 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-26T21:47:48.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.207 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:48.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.333 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:48.433 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:48.434 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:47:48.434 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:47:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:47:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:47:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:47:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:47:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T21:47:48.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:48.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:48.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:48.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:47:48.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:48.838 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:48.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:48.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:48.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:49.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:49.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.091 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:49.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:49.215 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:49.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:49.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[40587]: pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[40587]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:49.416 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[29406]: pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:49.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:47:49.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:47:49.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:49 smithi055 ceph-mon[29406]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:47:49.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:49.473 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:49 smithi067 ceph-mon[33346]: pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:47:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3424243472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:47:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3544406835' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T21:47:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:49 smithi067 ceph-mon[33346]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T21:47:49.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:49.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.726 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:49.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T21:47:49.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:49.856 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:49.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:49.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:49.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:50.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.109 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:50.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:50.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:50.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:50.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.364 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:50.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:50.366 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:50 smithi055 ceph-mon[40587]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:47:50.366 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:50 smithi055 ceph-mon[29406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:47:50.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:50.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:50 smithi067 ceph-mon[33346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T21:47:50.489 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:50.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.616 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:50.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:50.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:50.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:50.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:50.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:50.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:51.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:51.129 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147831 9.8 9:18de0198:::benchmark_data_smithi055_135863_object17180:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 405.514930913, 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0240032659999998, 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.176 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147831 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666342+0000", 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666343+0000", 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4470000000000001e-06 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666347+0000", 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6449999999999998e-06 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666403+0000", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6029000000000002e-05 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.666414+0000", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1065e-05 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.675183+0000", 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087694560000000001 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.675195+0000", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1766999999999999e-05 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661090+0000", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98589458299999999 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661176+0000", 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6285999999999993e-05 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.688687+0000", 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0275108209999999 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.690345+0000", 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016581670000000001 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147849 9.8 9:137978a5:::benchmark_data_smithi055_135863_object17198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:47:51.181 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 405.48951854199998, 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.011147448, 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147849 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:47:51.182 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691754+0000", 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691756+0000", 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.652e-06 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691764+0000", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1960000000000008e-06 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691784+0000", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9714000000000001e-05 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.691795+0000", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1756999999999999e-05 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692479+0000", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00068378600000000003 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.692492+0000", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2384e-05 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661266+0000", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.968774358 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661276+0000", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9159999999999996e-06 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.691612+0000", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0303359299999999 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.692227+0000", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00061501800000000005 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:47:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.692352+0000", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012515599999999999 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702314+0000", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0099620539999999997 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702361+0000", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6623999999999999e-05 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702557+0000", 2024-03-26T21:47:51.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019608900000000001 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702766+0000", 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000209464 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702902+0000", 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013535000000000001 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.189 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:147857 9.8 9:1c81c588:::benchmark_data_smithi055_135863_object17206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:47:51.189 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 405.46548550900002, 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.990900015, 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 147857 2024-03-26T21:47:51.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715787+0000", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715789+0000", 2024-03-26T21:47:51.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.759e-06 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715796+0000", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.883e-06 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715810+0000", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4405e-05 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715820+0000", 2024-03-26T21:47:51.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.781e-06 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715895+0000", 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4951999999999997e-05 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.715906+0000", 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.063e-05 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661303+0000", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94539728300000003 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.661311+0000", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5790000000000004e-06 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.695645+0000", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0343331120000001 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.695785+0000", 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014065 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.696089+0000", 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030426299999999998 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702539+0000", 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0064498619999999998 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.195 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.702910+0000", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00037042599999999998 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706506+0000", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0035965770000000001 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706567+0000", 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0748000000000002e-05 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.706687+0000", 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120105 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166764 9.8 9:14fe2036:::benchmark_data_smithi055_135863_object36113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 386.59600184599998, 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.166361408, 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:47:51.197 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166764 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585271+0000", 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585272+0000", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4699999999999999e-06 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585278+0000", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7960000000000001e-06 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585291+0000", 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3303e-05 2024-03-26T21:47:51.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585300+0000", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4849999999999992e-06 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585900+0000", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00059990599999999999 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.585980+0000", 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0327000000000005e-05 2024-03-26T21:47:51.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.586074+0000", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4301000000000004e-05 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.590001+0000", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0039261749999999996 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.604749+0000", 2024-03-26T21:47:51.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.014748618999999999 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.604764+0000", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4477e-05 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751549+0000", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.146784848 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751632+0000", 2024-03-26T21:47:51.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3701000000000004e-05 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166778 9.8 9:1f03b6ad:::benchmark_data_smithi055_135863_object36127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 386.56124877500002, 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.139688762, 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.203 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166778 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620024+0000", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620025+0000", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.53e-06 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620033+0000", 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1550000000000001e-06 2024-03-26T21:47:51.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620047+0000", 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4883e-05 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620057+0000", 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6250000000000002e-06 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620136+0000", 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8548999999999996e-05 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:24.620159+0000", 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3354e-05 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.751880+0000", 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1317214899999999 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752004+0000", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012312299999999999 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.752153+0000", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014930600000000001 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758277+0000", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0061243060000000004 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758655+0000", 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00037819600000000002 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.758697+0000", 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2063e-05 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759530+0000", 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00083214400000000005 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759595+0000", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4970999999999998e-05 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.759713+0000", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011806700000000001 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166798 9.5 9:a9aae264:::benchmark_data_smithi055_135863_object36147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:47:51.210 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 385.42613746799998, 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99086638599999999, 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166798 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755135+0000", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755137+0000", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.635e-06 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755142+0000", 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.921e-06 2024-03-26T21:47:51.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755167+0000", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4850000000000001e-05 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755177+0000", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0069e-05 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755250+0000", 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3651000000000003e-05 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.755263+0000", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2627e-05 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733027+0000", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97776415900000002 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733159+0000", 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013211200000000001 2024-03-26T21:47:51.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.733292+0000", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013281999999999999 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.738986+0000", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056941980000000001 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742091+0000", 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003104711 2024-03-26T21:47:51.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.745825+0000", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037338330000000002 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.745889+0000", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4539e-05 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746002+0000", 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112261 2024-03-26T21:47:51.216 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:166806 9.5 9:ab01a6aa:::benchmark_data_smithi055_135863_object36155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 385.41156009299999, 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97653648699999995, 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.217 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 166806 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769713+0000", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769714+0000", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3990000000000001e-06 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769721+0000", 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5719999999999999e-06 2024-03-26T21:47:51.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769734+0000", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3638000000000001e-05 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769743+0000", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4149999999999999e-06 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769817+0000", 2024-03-26T21:47:51.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4548999999999994e-05 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.769829+0000", 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.166e-05 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736190+0000", 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96636086799999998 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736300+0000", 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011020599999999999 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.736415+0000", 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115605 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742108+0000", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0056924000000000002 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.742327+0000", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00021956800000000001 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746057+0000", 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0037296909999999998 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.227 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746114+0000", 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6436e-05 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.746249+0000", 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013548 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241352 9.8 9:1a77692d:::benchmark_data_smithi055_135863_object110701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.228 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.30482267100001, 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96528207899999996, 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241352 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:47:51.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876452+0000", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1229999999999998e-06 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876461+0000", 2024-03-26T21:47:51.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9670000000000007e-06 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876481+0000", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0361e-05 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876493+0000", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.169e-05 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876566+0000", 2024-03-26T21:47:51.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3020000000000002e-05 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876576+0000", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9399999999999997e-06 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836333+0000", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.95975673500000003 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836431+0000", 2024-03-26T21:47:51.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8459e-05 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836511+0000", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0210999999999995e-05 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841566+0000", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050547559999999997 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841607+0000", 2024-03-26T21:47:51.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1267999999999999e-05 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841648+0000", 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0703e-05 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.234 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841697+0000", 2024-03-26T21:47:51.235 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:51 smithi055 ceph-mon[40587]: pgmap v917: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:47:51.235 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:51 smithi055 ceph-mon[29406]: pgmap v917: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:47:51.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9001000000000001e-05 2024-03-26T21:47:51.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841732+0000", 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4845000000000003e-05 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241370 9.8 9:1a2077f5:::benchmark_data_smithi055_135863_object110719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.282560563, 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94483976700000005, 2024-03-26T21:47:51.237 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.238 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.238 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.238 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.238 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241370 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:47:51.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898714+0000", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.621e-06 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898721+0000", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3529999999999997e-06 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898737+0000", 2024-03-26T21:47:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6283e-05 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898747+0000", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2979999999999997e-06 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898812+0000", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5160000000000006e-05 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898823+0000", 2024-03-26T21:47:51.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0910000000000001e-05 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838884+0000", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94006149800000005 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838956+0000", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2070999999999995e-05 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.839026+0000", 2024-03-26T21:47:51.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9647999999999995e-05 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841779+0000", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0027533039999999998 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843421+0000", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016415379999999999 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843462+0000", 2024-03-26T21:47:51.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0815999999999997e-05 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843511+0000", 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9166999999999999e-05 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843552+0000", 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1100000000000003e-05 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241386 9.8 9:15e63956:::benchmark_data_smithi055_135863_object110735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.26628437300002, 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92876690900000003, 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241386 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914989+0000", 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1060000000000001e-06 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914994+0000", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.04e-06 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915006+0000", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1435e-05 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915014+0000", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7619999999999995e-06 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915071+0000", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7256000000000002e-05 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915079+0000", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3780000000000001e-06 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840207+0000", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92512746099999998 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840287+0000", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0653000000000002e-05 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840368+0000", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0980000000000001e-05 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843430+0000", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030614000000000001 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843613+0000", 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000182805 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843672+0000", 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9079000000000001e-05 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843722+0000", 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0284000000000002e-05 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843755+0000", 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3269999999999998e-05 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241396 9.8 9:15ab1784:::benchmark_data_smithi055_135863_object110745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.25987417699997, 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.928642615, 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.251 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241396 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921399+0000", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.032e-06 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921405+0000", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0209999999999999e-06 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921417+0000", 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2874000000000001e-05 2024-03-26T21:47:51.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921426+0000", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1429999999999998e-06 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921482+0000", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6236999999999997e-05 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921490+0000", 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9610000000000005e-06 2024-03-26T21:47:51.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841812+0000", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92032255900000004 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841909+0000", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7065999999999995e-05 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.842006+0000", 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6650999999999993e-05 2024-03-26T21:47:51.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845746+0000", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003739839 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845805+0000", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9500000000000003e-05 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.849965+0000", 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0041593410000000004 2024-03-26T21:47:51.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850001+0000", 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6341000000000003e-05 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850041+0000", 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0049999999999998e-05 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.257 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241442 9.8 9:117f2a93:::benchmark_data_smithi055_135863_object110791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.22356633300001, 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90324378800000005, 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241442 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.258 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957708+0000", 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.956e-06 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.259 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957726+0000", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8070000000000001e-05 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957742+0000", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5252000000000001e-05 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957750+0000", 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1359999999999997e-06 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.260 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957831+0000", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1278000000000003e-05 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957843+0000", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2038999999999999e-05 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843789+0000", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88594599600000001 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.261 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843888+0000", 2024-03-26T21:47:51.262 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8515000000000001e-05 2024-03-26T21:47:51.262 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.262 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.262 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843989+0000", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000101331 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852756+0000", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087672730000000008 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852772+0000", 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5968999999999999e-05 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.263 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860849+0000", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0080765820000000006 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860901+0000", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2301e-05 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860950+0000", 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9089999999999999e-05 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.264 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241445 9.8 9:195f423d:::benchmark_data_smithi055_135863_object110794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.22257255099998, 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.902364579, 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241445 2024-03-26T21:47:51.265 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.266 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958701+0000", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4399999999999999e-07 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958706+0000", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.091e-06 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958721+0000", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4491999999999999e-05 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.267 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958729+0000", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.541e-06 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958812+0000", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2684999999999999e-05 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958822+0000", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0095000000000001e-05 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.268 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845839+0000", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88701723600000004 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845945+0000", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105958 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.846050+0000", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104899 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.269 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852779+0000", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0067287370000000003 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852812+0000", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3673000000000001e-05 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860989+0000", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081766100000000008 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861031+0000", 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1711000000000001e-05 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861065+0000", 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4007000000000002e-05 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241482 9.8 9:1973c4a0:::benchmark_data_smithi055_135863_object110831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 301.19373555200002, 2024-03-26T21:47:51.271 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88112820999999997, 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241482 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.272 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987539+0000", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5999999999999999e-06 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987545+0000", 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2820000000000002e-06 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.273 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987558+0000", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2744e-05 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987566+0000", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1259999999999998e-06 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987642+0000", 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6149999999999994e-05 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.274 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987652+0000", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0351e-05 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847815+0000", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.86016230900000001 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847915+0000", 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000100707 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.275 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847989+0000", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3889999999999999e-05 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860773+0000", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012783341 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861101+0000", 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032820899999999999 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.276 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868358+0000", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072569499999999999 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868618+0000", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025993400000000002 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868665+0000", 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7617000000000002e-05 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.277 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:52100 10.8 10:13e4aa23:::benchmark_data_smithi055_163313_object52099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 226.84600549000001, 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.023603611, 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.278 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 52100 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.279 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335269+0000", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.57e-06 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335299+0000", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0700000000000001e-05 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335311+0000", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1131000000000001e-05 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.280 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335318+0000", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9109999999999993e-06 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335557+0000", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023892 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335563+0000", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8309999999999997e-06 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.281 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.463863+0000", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.128299688 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.463877+0000", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3730999999999999e-05 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.358791+0000", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.89491480700000003 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.282 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.358871+0000", 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9321999999999997e-05 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:52111 10.4 10:26e68298:::benchmark_data_smithi055_163313_object52110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 226.66885097100001, 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.85519976200000003, 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.283 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 52111 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.284 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512423+0000", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6899999999999999e-06 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512448+0000", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4522999999999999e-05 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512465+0000", 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7441000000000001e-05 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.285 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512476+0000", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.029e-05 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512544+0000", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8879000000000002e-05 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512556+0000", 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1127e-05 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.286 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.356974+0000", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84441795200000003 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.357121+0000", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014747100000000001 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.357306+0000", 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000184475 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.287 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.361403+0000", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040972389999999999 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.365943+0000", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004539838 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367325+0000", 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001382716 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.288 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367388+0000", 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3091999999999998e-05 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367621+0000", 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023302899999999999 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164765 10.20 10:055a7cd5:::benchmark_data_smithi055_163313_object60716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:47:51.289 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 70.216100444000006, 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91669290199999998, 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164765 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.290 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965174+0000", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.499e-06 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.291 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965180+0000", 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0190000000000003e-06 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965192+0000", 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2476999999999999e-05 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965201+0000", 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0960000000000007e-06 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.292 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965298+0000", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6576999999999994e-05 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965309+0000", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1460999999999999e-05 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877721+0000", 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91241161400000004 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.293 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877812+0000", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1153e-05 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877996+0000", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018388599999999999 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881439+0000", 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034433939999999998 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.294 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881533+0000", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3682000000000002e-05 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881763+0000", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000229497 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881821+0000", 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7992999999999999e-05 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.295 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881865+0000", 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4554000000000001e-05 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164781 10.20 10:0451c1db:::benchmark_data_smithi055_163313_object60732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 70.203926916, 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90551289400000001, 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.296 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164781 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.297 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977347+0000", 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3689999999999999e-06 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977352+0000", 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4139999999999996e-06 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.298 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977361+0000", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1390000000000001e-06 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977367+0000", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7159999999999997e-06 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977845+0000", 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00047740400000000001 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.299 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977855+0000", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0068000000000001e-05 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879295+0000", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90144047199999999 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879405+0000", 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000110192 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.300 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879593+0000", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000187982 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881647+0000", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020535129999999999 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881694+0000", 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6876999999999999e-05 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.301 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882794+0000", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001100209 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882830+0000", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5611000000000002e-05 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882859+0000", 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.8928000000000002e-05 2024-03-26T21:47:51.302 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164858 10.20 10:05bf36b7:::benchmark_data_smithi055_163313_object60809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 70.150028233, 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.85176363099999997, 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.303 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164858 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.304 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031246+0000", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.522e-06 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031253+0000", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6549999999999998e-06 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031274+0000", 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1809000000000001e-05 2024-03-26T21:47:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031283+0000", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7630000000000006e-06 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031350+0000", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6995000000000005e-05 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031360+0000", 2024-03-26T21:47:51.306 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6830000000000003e-06 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880403+0000", 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84904316999999996 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880515+0000", 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112048 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:47:51.307 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880682+0000", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000167128 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881880+0000", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001197353 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882909+0000", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0010296680000000001 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.308 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882957+0000", 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8137999999999999e-05 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882985+0000", 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.758e-05 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.883008+0000", 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3119e-05 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164882 10.20 10:06434657:::benchmark_data_smithi055_163313_object60833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 70.135514817000001, 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84022213000000001, 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164882 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:47:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045759+0000", 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0210000000000001e-06 2024-03-26T21:47:51.311 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045764+0000", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.327e-06 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045774+0000", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9920000000000002e-06 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045782+0000", 2024-03-26T21:47:51.312 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9799999999999998e-06 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045984+0000", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00020188699999999999 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045995+0000", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1035e-05 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881926+0000", 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.83593075800000005 2024-03-26T21:47:51.313 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882029+0000", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103244 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882173+0000", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000144365 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.883023+0000", 2024-03-26T21:47:51.314 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00084954200000000001 2024-03-26T21:47:51.315 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.318 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.318 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.318 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885833+0000", 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028101329999999998 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885884+0000", 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1007999999999997e-05 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885936+0000", 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1517000000000002e-05 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:47:51.319 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885980+0000", 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4320999999999998e-05 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:47:51.320 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:47:51.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:51.321 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:51 smithi067 ceph-mon[33346]: pgmap v917: 148 pgs: 148 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:47:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.349 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:51.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:51.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:51.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:51.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:51.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:51.676 DEBUG:teuthology.orchestra.run.smithi055:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-26T21:47:51.679 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T21:47:51.679 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T21:47:51.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:51.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:51.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:51.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:51.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:hints = 1 2024-03-26T21:47:51.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T21:47:51.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object prefix: benchmark_data_smithi055_186849 2024-03-26T21:47:51.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:47:51.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 0 0 0 0 0 0 - 0 2024-03-26T21:47:51.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:51.999 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:52.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:52.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:52.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:52.277 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:52.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:52.394 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:52.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.560 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:52.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:52.662 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:52.703 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:52.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:52.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:52.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 1 15 3482 3467 216.676 216.688 0.011956 0.00458948 2024-03-26T21:47:52.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:52.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:52.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:47:53.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:53.125 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:53.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:53.231 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:47:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:52 smithi067 ceph-mon[33346]: pgmap v918: 148 pgs: 148 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:47:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:52 smithi067 ceph-mon[33346]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:47:53.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:52 smithi055 ceph-mon[29406]: pgmap v918: 148 pgs: 148 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:47:53.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:52 smithi055 ceph-mon[29406]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:47:53.240 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:52 smithi055 ceph-mon[40587]: pgmap v918: 148 pgs: 148 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:47:53.240 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:52 smithi055 ceph-mon[40587]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T21:47:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:53.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:53.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.540 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:53.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:53.679 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:53.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:53.814 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:53.891 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 2 15 6601 6586 205.799 194.938 0.0060176 0.00485017 2024-03-26T21:47:53.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:53.955 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:54.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:54.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:54.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:54.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:54.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:54.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:54.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:54.777 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:54.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:54.895 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 3 16 9743 9727 202.367 196.312 0.00330971 0.00492932 2024-03-26T21:47:54.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:54.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:55.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:55.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:54 smithi055 ceph-mon[29406]: pgmap v920: 148 pgs: 148 active+clean; 162 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 380 op/s 2024-03-26T21:47:55.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:54 smithi055 ceph-mon[40587]: pgmap v920: 148 pgs: 148 active+clean; 162 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 380 op/s 2024-03-26T21:47:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:55.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.199 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:55.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:54 smithi067 ceph-mon[33346]: pgmap v920: 148 pgs: 148 active+clean; 162 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 380 op/s 2024-03-26T21:47:55.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:55.341 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:55.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:55.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:55.612 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:55.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:55.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:55.889 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:55.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:55.895 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 4 16 12209 12193 190.315 154.125 0.00881389 0.00524756 2024-03-26T21:47:55.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:56.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:56.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:56.157 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:47:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:56.203 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:47:56.243 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:47:56.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:56.290 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:47:56.327 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:56.332 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:47:56.373 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:47:56.411 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:47:56.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:56.454 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:47:56.468 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:47:56.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:56.579 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:56.727 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.769 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:56.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:56.896 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 5 16 14579 14563 181.85 148.125 0.00416377 0.00548986 2024-03-26T21:47:56.913 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:56.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:56.923 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:47:57.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:57.060 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:57.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:56 smithi067 ceph-mon[33346]: pgmap v921: 148 pgs: 148 active+clean; 188 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 376 op/s 2024-03-26T21:47:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1624144248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:57.261 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[29406]: pgmap v921: 148 pgs: 148 active+clean; 188 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 376 op/s 2024-03-26T21:47:57.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1624144248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:57.262 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:57.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[40587]: pgmap v921: 148 pgs: 148 active+clean; 188 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 23 MiB/s wr, 376 op/s 2024-03-26T21:47:57.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1624144248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:47:57.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:47:57.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:57.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:57.470 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:57.572 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.615 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:57.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:57.716 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.755 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:57.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:57.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:47:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:57.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:47:57.895 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:47:57.897 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 6 16 17253 17237 179.364 167.125 0.00670676 0.00556842 2024-03-26T21:47:57.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:57.970 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[29406]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[40587]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:47:57.971 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:57 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:47:57.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.039 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:58.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:58.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:58.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:47:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2965994237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:47:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-mon[33346]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T21:47:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:47:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:57 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:47:58.274 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:58.312 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:58.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:58.456 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:58.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.596 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:58.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:58.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:47:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.738 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:58.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:47:58.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:47:58.882 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:58.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:47:58.897 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 7 16 19604 19588 174.734 146.938 0.00622791 0.00571575 2024-03-26T21:47:58.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:47:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:59.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:47:59.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:58 smithi067 ceph-mon[33346]: pgmap v923: 148 pgs: 148 active+clean; 519 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T21:47:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:58 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:47:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:58 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:47:59.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[40587]: pgmap v923: 148 pgs: 148 active+clean; 519 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T21:47:59.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[29406]: pgmap v923: 148 pgs: 148 active+clean; 519 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:47:59.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:47:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:47:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:47:59.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:47:59.453 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:47:59.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:47:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.590 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:59.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:47:59.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:47:59.734 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:47:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:47:59.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:47:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:47:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:47:59.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:47:59.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 8 16 22024 22008 171.799 151.25 0.00391774 0.00581412 2024-03-26T21:47:59.957 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T21:47:59.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:00.111 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:00.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:48:00.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:48:00.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:48:00.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3964732169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:48:00.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:47:59 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:48:00.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:48:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3964732169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T21:48:00.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3964732169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:48:00.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:48:00.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:47:59 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:48:00.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:00.433 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:00.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:00.578 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:00.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.724 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:00.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:00.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:00.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:00.897 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 9 16 24159 24143 167.538 133.438 0.00787793 0.00596296 2024-03-26T21:48:00.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.048 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:01.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:01.149 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.189 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:01.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: pgmap v925: 148 pgs: 148 active+clean; 1.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:48:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:00 smithi067 ceph-mon[33346]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:48:01.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:01.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: pgmap v925: 148 pgs: 148 active+clean; 1.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[40587]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:48:01.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: pgmap v925: 148 pgs: 148 active+clean; 1.0 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-26T21:48:01.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:48:01.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:00 smithi055 ceph-mon[29406]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T21:48:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:01.372 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:48:01.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.464 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:01.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:01.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:01.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:01.689 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:01.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:01.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:01.873 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:01.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:01.897 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 10 16 26210 26194 163.598 128.188 0.00985799 0.0061047 2024-03-26T21:48:01.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:02.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:02.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:48:02.133 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[29406]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:48:02.134 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:01 smithi055 ceph-mon[40587]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:48:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:48:02.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:01 smithi067 ceph-mon[33346]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T21:48:02.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:02.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:02.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:02.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:02.649 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:02.690 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:02.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:48:02.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:02.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 11 16 28325 28309 160.743 132.188 0.00234873 0.00619889 2024-03-26T21:48:02.927 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:02.971 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:02.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: pgmap v927: 148 pgs: 148 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.41k op/s 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:03.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: pgmap v927: 148 pgs: 148 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.41k op/s 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:03.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:02 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:03.072 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.120 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:03.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:03.221 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:48:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:48:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: pgmap v927: 148 pgs: 148 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.41k op/s 2024-03-26T21:48:03.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:48:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:48:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:48:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:48:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:03.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:03.262 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:03.364 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.407 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:03.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:03.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T21:48:03.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:03.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:03.689 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:03.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:03.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 12 16 30266 30250 157.459 121.312 0.0048162 0.00634327 2024-03-26T21:48:03.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:03.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:03.960 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:04.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:48:04.116 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:48:04.117 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:48:04.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:04.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:04.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:48:04.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:48:04.119 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:48:04.120 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:03 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:48:04.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:48:04.243 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:48:04.244 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:03 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:48:04.317 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:04.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:04.459 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:04.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:04.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:04.730 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:04.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:04.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:04.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 13 16 32013 31997 153.742 109.188 0.00414075 0.00649543 2024-03-26T21:48:04.916 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:04.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:05.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:05.057 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:05.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: pgmap v930: 148 pgs: 148 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 3.11 scrub starts 2024-03-26T21:48:05.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 3.11 scrub ok 2024-03-26T21:48:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:48:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:48:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:48:05.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:04 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:48:05.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:05.311 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:48:05.311 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:48:05.311 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: pgmap v930: 148 pgs: 148 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-03-26T21:48:05.311 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:48:05.311 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 3.11 scrub starts 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 3.11 scrub ok 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: pgmap v930: 148 pgs: 148 active+clean; 1.7 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-03-26T21:48:05.312 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 3.11 scrub starts 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 3.11 scrub ok 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:48:05.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:04 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:48:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:05.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:05.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:05.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:05.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:05.823 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:05.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 14 16 33678 33662 150.194 104.062 0.00756895 0.00665237 2024-03-26T21:48:05.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:05.964 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:05.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:06.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:06.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:06.207 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:06.229 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:48:06.229 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:48:06.245 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:48:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:48:06.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:05 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:48:06.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:06 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:48:06.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:06.385 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:06.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:48:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:06 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:48:06.526 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:06.628 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:06.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:06.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:06.801 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:06.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 15 16 35473 35457 147.661 112.188 0.00591374 0.00676555 2024-03-26T21:48:06.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:06.934 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:06.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:07.036 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:07.077 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[40587]: pgmap v931: 148 pgs: 148 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[40587]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[29406]: pgmap v931: 148 pgs: 148 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-03-26T21:48:07.136 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:07 smithi055 ceph-mon[29406]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:48:07.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:07.218 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:07.320 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:07.361 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:07.463 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:07 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:48:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:07 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:48:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:07 smithi067 ceph-mon[33346]: pgmap v931: 148 pgs: 148 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-03-26T21:48:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:07 smithi067 ceph-mon[33346]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T21:48:07.506 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:07.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:07.639 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:07.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.781 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:07.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:07.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:07.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 16 16 37393 37377 145.932 120 0.00816011 0.00684684 2024-03-26T21:48:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:07.918 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:07.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:07.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:08.019 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:08.059 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:08.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:48:08.266 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:48:08.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:48:08.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:08.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:08.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:48:08.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:08 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:48:08.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:08.613 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:08.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:08.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:08.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:08.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:08.898 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 17 16 39241 39225 144.142 115.5 0.00979429 0.00693221 2024-03-26T21:48:08.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.041 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:09.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:09.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:09.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:09.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:09.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[29406]: pgmap v933: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T21:48:09.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:48:09.241 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:48:09.241 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:09.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:09.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[40587]: pgmap v933: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T21:48:09.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:48:09.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:09 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:48:09.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:09.292 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:48:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:09.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:09.316 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:09 smithi067 ceph-mon[33346]: pgmap v933: 148 pgs: 148 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:09 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:48:09.392 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:09 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:48:09.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:09.452 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:09.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:09.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:09.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:09.729 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:09.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:09.878 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:09.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:09.899 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 18 16 41006 40990 142.257 110.312 0.00873762 0.0070232 2024-03-26T21:48:09.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:10.023 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:10.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.158 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:10.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:10.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:10.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:10.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:48:10.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:48:10.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:48:10.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:48:10.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:10 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:48:10.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.446 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:10.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:48:10.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:10 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:48:10.547 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:10.586 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:10.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:10.725 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:10.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.863 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:10.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:10.899 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 19 16 42750 42734 140.507 109 0.00942222 0.00711165 2024-03-26T21:48:10.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:10.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:11.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:11.111 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:11.148 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:11.217 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:48:11.217 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[29406]: pgmap v934: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T21:48:11.217 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:48:11.218 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:48:11.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[40587]: pgmap v934: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T21:48:11.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:48:11.218 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:11 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:48:11.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:11.260 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:48:11.283 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:11.304 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:48:11.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:48:11.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:11.387 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:48:11.413 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:11.434 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:48:11.484 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:48:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:11 smithi067 ceph-mon[33346]: pgmap v934: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.51k op/s 2024-03-26T21:48:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:11 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:48:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:11 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:48:11.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:11.527 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:48:11.532 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:48:11.532 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:48:11.532 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:48:11.532 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:48:11.532 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:48:11.533 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:48:11.533 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:48:11.533 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:48:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:11.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:11.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:11.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:11.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:11.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:48:11.908007+0000 min lat: 0.00133566 max lat: 0.179135 avg lat: 0.00720777 2024-03-26T21:48:11.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:48:11.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 20 16 44390 44374 138.607 102.5 0.0085484 0.00720777 2024-03-26T21:48:11.924 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:11.967 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:11.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:12.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:12.110 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:12.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:12.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:48:12.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:48:12.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3820781238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3820781238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:12.278 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:48:12.279 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:12 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:48:12.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:12.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:12.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3820781238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:48:12.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:12 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:48:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:12.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:12.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.668 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:12.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:12.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:12.806 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:12.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 21 16 45974 45958 136.721 99 0.0132366 0.00730931 2024-03-26T21:48:12.908 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:12.932 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:12.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:13.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:13.071 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:13.173 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:13.214 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: pgmap v935: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:48:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: pgmap v935: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:48:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:13 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:48:13.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:13.358 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:13.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.484 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: pgmap v935: 148 pgs: 148 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:48:13.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:13 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:48:13.585 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:13.625 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:13.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:13.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:13.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:13.893 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:13.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:13.901 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 22 16 47233 47217 134.084 78.6875 0.00450083 0.00745254 2024-03-26T21:48:13.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:14.035 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:14.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:14.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:14.267 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.306 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:14.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:14.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:48:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:48:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:48:14.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:48:14.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:48:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:48:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:48:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:14 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:48:14.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:14.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:14 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:48:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:14 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:48:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:14 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:48:14.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:14 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:48:14.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.574 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:14.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:14.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:14.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:14.844 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:14.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 23 16 48790 48774 132.485 97.3125 0.00214316 0.00753598 2024-03-26T21:48:14.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:14.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:15.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:15.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:15.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:15.241 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:15.343 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:15.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:15.482 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:48:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:15 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:48:15.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:15.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T21:48:15.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:48:15.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:48:15.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:48:15.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:48:15.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:48:15.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:48:15.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:48:15.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:15 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:48:15.623 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:15.664 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:15.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:15.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:15.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 24 16 50291 50275 130.874 93.8125 0.00741074 0.00763618 2024-03-26T21:48:15.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:15.947 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:15.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:16.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:16.085 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:16.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.233 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:16.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:16.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:16.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:16 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:48:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:16 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:48:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:16 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:48:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:16 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:48:16.515 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:48:16.536 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:48:16.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:48:16.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:16 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:48:16.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:16.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:16.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:16.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:16.900 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 25 15 51888 51873 129.634 99.875 0.00792893 0.00770838 2024-03-26T21:48:16.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:16.939 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:16.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:17.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:17.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:17.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:17.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:17.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:17.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: pgmap v937: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:48:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:48:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:48:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:48:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:48:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:48:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:48:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:48:17.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:48:17.490 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:17.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: pgmap v937: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: pgmap v937: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T21:48:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:48:17.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:17 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:48:17.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:17.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:17.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.774 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:17.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:17.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:17.901 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 26 16 53395 53379 128.266 94.125 0.00499163 0.0077901 2024-03-26T21:48:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:17.917 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:17.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:17.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:18.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:18.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:18.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:18.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:18.328 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:18.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:18.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:18.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:18 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:48:18.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:18 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:48:18.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:18.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:18 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:48:18.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:18 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:48:18.546 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:48:18.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:48:18.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:18.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:48:18.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:48:18.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:18.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:18.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:18.769 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:18.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:18.901 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 27 15 54894 54879 126.985 93.75 0.0131335 0.0078694 2024-03-26T21:48:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:18.917 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:18.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:19.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:19.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:19.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:19.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:19.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[40587]: pgmap v938: 148 pgs: 148 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 945 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[29406]: pgmap v938: 148 pgs: 148 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 945 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:19 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:48:19.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:19.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:19.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:19.481 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:19 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:19 smithi067 ceph-mon[33346]: pgmap v938: 148 pgs: 148 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 945 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T21:48:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:19 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:48:19.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:19 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:48:19.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:19.616 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:19.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:19.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:19.852 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:19.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:48:19.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 28 16 56241 56225 125.452 84.125 0.0169167 0.0079626 2024-03-26T21:48:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:19.906 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:19.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:19.973 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:48:19.974 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:48:19.974 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:48:19.974 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:48:19.974 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:48:20.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:20.114 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T21:48:20.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:20.211 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:20.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:48:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:48:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:48:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:48:20.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:48:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:48:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:48:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:48:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:48:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:20 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:48:20.340 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:20.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:20.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:48:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:20 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:48:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:20 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:48:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:20 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:48:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:20 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:48:20.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:20.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:20.710 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:20.757 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:20.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:20.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:20.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:20.902 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 29 16 57581 57565 124.015 83.75 0.00945159 0.00805818 2024-03-26T21:48:20.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:21.043 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:21.145 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:21.184 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:21.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:21.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: pgmap v939: 148 pgs: 148 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: pgmap v939: 148 pgs: 148 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:48:21.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:48:21.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:21 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:48:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.327 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:21.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:21.428 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.477 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:21.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: pgmap v939: 148 pgs: 148 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:48:21.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4154470515' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:48:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T21:48:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:48:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:21 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:48:21.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:21.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:21.729 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:48:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:21.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:21.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:21.885 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:21.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:21.896 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:48:21.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 30 16 58873 58857 122.565 80.75 0.0153912 0.00815069 2024-03-26T21:48:21.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.021 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:22.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:22.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:22.156 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:22.259 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.302 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:22.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:48:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: 3.8 scrub starts 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: 3.8 scrub ok 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[40587]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: 3.8 scrub starts 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: 3.8 scrub ok 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:48:22.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:48:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:22 smithi055 ceph-mon[29406]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:48:22.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:22.450 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T21:48:22.450 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:48:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:48:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: 3.8 scrub starts 2024-03-26T21:48:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: 3.8 scrub ok 2024-03-26T21:48:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:48:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:48:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1128239194' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:48:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:22 smithi067 ceph-mon[33346]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T21:48:22.542 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.595 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:22.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:22.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:22.724 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:22.780 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:22.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:22.856 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:22.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:22.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 31 16 60138 60122 121.162 79.0625 0.00935013 0.00824832 2024-03-26T21:48:22.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:23.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:23.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:23.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:23.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:23.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:23.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:23.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:23 smithi067 ceph-mon[33346]: pgmap v941: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-26T21:48:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:23 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:48:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:23 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:48:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:23 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:48:23.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:23 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:48:23.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[40587]: pgmap v941: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-26T21:48:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:48:23.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[29406]: pgmap v941: 148 pgs: 148 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:48:23.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:23 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:48:23.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:23.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:23.826 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:23.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 32 16 61569 61553 120.171 89.4375 0.0163309 0.00831622 2024-03-26T21:48:23.929 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:23.968 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:23.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:24.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:24.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:24.235 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 4.5 scrub starts 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 4.5 scrub ok 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:48:24.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 4.5 scrub starts 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 4.5 scrub ok 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:48:24.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:24 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:48:24.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:24.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:24.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:48:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:48:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:24.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 4.5 scrub starts 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 4.5 scrub ok 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:48:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:24 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:48:24.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:24.608 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:24.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:24.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:24.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:24.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:24.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 33 16 62841 62825 118.938 79.5 0.0115934 0.0084026 2024-03-26T21:48:24.937 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:24.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:25.039 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:25.079 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:25.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:25.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:25.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:25.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[29406]: pgmap v944: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[29406]: 5.12 deep-scrub starts 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[29406]: 5.12 deep-scrub ok 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[40587]: pgmap v944: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[40587]: 5.12 deep-scrub starts 2024-03-26T21:48:25.446 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:25 smithi055 ceph-mon[40587]: 5.12 deep-scrub ok 2024-03-26T21:48:25.462 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:25 smithi067 ceph-mon[33346]: pgmap v944: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T21:48:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:25 smithi067 ceph-mon[33346]: 5.12 deep-scrub starts 2024-03-26T21:48:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:25 smithi067 ceph-mon[33346]: 5.12 deep-scrub ok 2024-03-26T21:48:25.493 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.550 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:25.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:25.564 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:25.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:25.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:25.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:25.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:25.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 34 16 64161 64145 117.867 82.5 0.0127996 0.00847978 2024-03-26T21:48:25.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:25.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:25.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:26.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:48:26.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:26.148 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:26.250 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.294 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:26.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:26.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:26 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:48:26.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:26 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:48:26.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:26 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:48:26.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:26 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:48:26.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:26.429 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:26 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:48:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:26 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:48:26.531 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:26.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:26.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.761 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:26.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:26.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:26.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:26.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:26.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 35 16 65443 65427 116.788 80.125 0.00665158 0.00855685 2024-03-26T21:48:27.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:27.042 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:27.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:27.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:27.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:27.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:27.466 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: pgmap v945: 148 pgs: 148 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: 5.11 deep-scrub starts 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: 5.11 deep-scrub ok 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:48:27.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-mon[33346]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:48:27.564 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: pgmap v945: 148 pgs: 148 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-26T21:48:27.564 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: 5.11 deep-scrub starts 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: 5.11 deep-scrub ok 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[40587]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: pgmap v945: 148 pgs: 148 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: 5.11 deep-scrub starts 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: 5.11 deep-scrub ok 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:48:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:27 smithi055 ceph-mon[29406]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T21:48:27.568 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:27.609 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:27.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:27.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:27.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:27.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:27.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 36 16 66660 66644 115.657 76.0625 0.00896794 0.00864085 2024-03-26T21:48:27.966 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:48:27.966 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:48:27.966 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:48:27.967 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:48:27.968 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:48:27.968 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:48:27.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:27.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:28.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:28.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:28.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:28.286 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:28.388 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:28.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1242182816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:48:28.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:48:28.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:28 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:48:28.503 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1242182816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:48:28.504 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1242182816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:28.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:48:28.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:48:28.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:48:28.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:48:28.533 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.589 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:28.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:28.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:28.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:28.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:28.870 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:28.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:28.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 37 16 67891 67875 114.611 76.9375 0.00586658 0.00871961 2024-03-26T21:48:28.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:29.003 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:29.034 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:29.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:29.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:29.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[29406]: pgmap v947: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[40587]: pgmap v947: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:48:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:29 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:48:29.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:29.408 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:48:29.419 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:48:29.435 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:29 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:29 smithi067 ceph-mon[33346]: pgmap v947: 148 pgs: 148 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T21:48:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:29 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:48:29.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:29 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:48:29.537 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:29.577 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:29.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:29.720 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:29.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:29.863 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:29.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:29.904 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 38 16 69136 69120 113.643 77.8125 0.00643678 0.00879498 2024-03-26T21:48:29.964 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:30.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.006 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:30.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:30.146 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:30.213 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":947,"stamp":"2024-03-26T21:48:27.726178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4090173864,"num_objects":62600,"num_object_clones":0,"num_object_copies":187800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62600,"num_whiteouts":0,"num_read":2384,"num_read_kb":2232,"num_write":63303,"num_write_kb":3996189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":1777738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63421,"ondisk_log_size":63421,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17590268,"kb_used_data":12005872,"kb_used_omap":20,"kb_used_meta":5584299,"kb_avail":732370948,"statfs":{"total":767960285184,"available":749947850752,"internally_reserved":0,"allocated":12294012928,"data_stored":12288928096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20884,"internal_metadata":5718322796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,9,8,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":576978944,"num_objects":8804,"num_object_clones":0,"num_object_copies":26412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8804,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8804,"num_write_kb":563456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001580"},"pg_stats":[{"pgid":"11.3","version":"449'4056","reported_seq":4100,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910956+0000","last_change":"2024-03-26T21:47:39.050006+0000","last_active":"2024-03-26T21:48:26.910956+0000","last_peered":"2024-03-26T21:48:26.910956+0000","last_clean":"2024-03-26T21:48:26.910956+0000","last_became_active":"2024-03-26T21:47:39.049657+0000","last_became_peered":"2024-03-26T21:47:39.049657+0000","last_unstale":"2024-03-26T21:48:26.910956+0000","last_undegraded":"2024-03-26T21:48:26.910956+0000","last_fullsized":"2024-03-26T21:48:26.910956+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.260841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265814016,"num_objects":4056,"num_object_clones":0,"num_object_copies":12168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":259584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":617,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.193361+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:48:23.193361+0000","last_peered":"2024-03-26T21:48:23.193361+0000","last_clean":"2024-03-26T21:48:23.193361+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:48:23.193361+0000","last_undegraded":"2024-03-26T21:48:23.193361+0000","last_fullsized":"2024-03-26T21:48:23.193361+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.420796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921794+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:48:21.921794+0000","last_peered":"2024-03-26T21:48:21.921794+0000","last_clean":"2024-03-26T21:48:21.921794+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:48:21.921794+0000","last_undegraded":"2024-03-26T21:48:21.921794+0000","last_fullsized":"2024-03-26T21:48:21.921794+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:27.604011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":1108,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896708+0000","last_change":"2024-03-26T21:48:03.066076+0000","last_active":"2024-03-26T21:48:26.896708+0000","last_peered":"2024-03-26T21:48:26.896708+0000","last_clean":"2024-03-26T21:48:26.896708+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:48:26.896708+0000","last_undegraded":"2024-03-26T21:48:26.896708+0000","last_fullsized":"2024-03-26T21:48:26.896708+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.029799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":536,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.194428+0000","last_change":"2024-03-26T21:47:25.630901+0000","last_active":"2024-03-26T21:48:23.194428+0000","last_peered":"2024-03-26T21:48:23.194428+0000","last_clean":"2024-03-26T21:48:23.194428+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:48:23.194428+0000","last_undegraded":"2024-03-26T21:48:23.194428+0000","last_fullsized":"2024-03-26T21:48:23.194428+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:48.426306+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"450'4162","reported_seq":4206,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:27.569237+0000","last_change":"2024-03-26T21:47:39.048799+0000","last_active":"2024-03-26T21:48:27.569237+0000","last_peered":"2024-03-26T21:48:27.569237+0000","last_clean":"2024-03-26T21:48:27.569237+0000","last_became_active":"2024-03-26T21:47:39.048454+0000","last_became_peered":"2024-03-26T21:47:39.048454+0000","last_unstale":"2024-03-26T21:48:27.569237+0000","last_undegraded":"2024-03-26T21:48:27.569237+0000","last_fullsized":"2024-03-26T21:48:27.569237+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272760832,"num_objects":4162,"num_object_clones":0,"num_object_copies":12486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":266368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":583,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.192907+0000","last_change":"2024-03-26T21:47:23.550040+0000","last_active":"2024-03-26T21:48:23.192907+0000","last_peered":"2024-03-26T21:48:23.192907+0000","last_clean":"2024-03-26T21:48:23.192907+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:48:23.192907+0000","last_undegraded":"2024-03-26T21:48:23.192907+0000","last_fullsized":"2024-03-26T21:48:23.192907+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:51.983719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":879,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911101+0000","last_change":"2024-03-26T21:47:24.976202+0000","last_active":"2024-03-26T21:48:26.911101+0000","last_peered":"2024-03-26T21:48:26.911101+0000","last_clean":"2024-03-26T21:48:26.911101+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:48:26.911101+0000","last_undegraded":"2024-03-26T21:48:26.911101+0000","last_fullsized":"2024-03-26T21:48:26.911101+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:53.571891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.912292+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:48:21.912292+0000","last_peered":"2024-03-26T21:48:21.912292+0000","last_clean":"2024-03-26T21:48:21.912292+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:48:21.912292+0000","last_undegraded":"2024-03-26T21:48:21.912292+0000","last_fullsized":"2024-03-26T21:48:21.912292+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:04.817836+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":878,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.898837+0000","last_change":"2024-03-26T21:48:05.290034+0000","last_active":"2024-03-26T21:48:26.898837+0000","last_peered":"2024-03-26T21:48:26.898837+0000","last_clean":"2024-03-26T21:48:26.898837+0000","last_became_active":"2024-03-26T21:44:49.410905+0000","last_became_peered":"2024-03-26T21:44:49.410905+0000","last_unstale":"2024-03-26T21:48:26.898837+0000","last_undegraded":"2024-03-26T21:48:26.898837+0000","last_fullsized":"2024-03-26T21:48:26.898837+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.592759+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"449'4063","reported_seq":4107,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911600+0000","last_change":"2024-03-26T21:47:39.047939+0000","last_active":"2024-03-26T21:48:26.911600+0000","last_peered":"2024-03-26T21:48:26.911600+0000","last_clean":"2024-03-26T21:48:26.911600+0000","last_became_active":"2024-03-26T21:47:39.047535+0000","last_became_peered":"2024-03-26T21:47:39.047535+0000","last_unstale":"2024-03-26T21:48:26.911600+0000","last_undegraded":"2024-03-26T21:48:26.911600+0000","last_fullsized":"2024-03-26T21:48:26.911600+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4063,"log_dups_size":0,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:47.698282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":12189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":260032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"426'24","reported_seq":565,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.194533+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:48:23.194533+0000","last_peered":"2024-03-26T21:48:23.194533+0000","last_clean":"2024-03-26T21:48:23.194533+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:48:23.194533+0000","last_undegraded":"2024-03-26T21:48:23.194533+0000","last_fullsized":"2024-03-26T21:48:23.194533+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:52.521281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":537,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191098+0000","last_change":"2024-03-26T21:47:15.544701+0000","last_active":"2024-03-26T21:48:23.191098+0000","last_peered":"2024-03-26T21:48:23.191098+0000","last_clean":"2024-03-26T21:48:23.191098+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:48:23.191098+0000","last_undegraded":"2024-03-26T21:48:23.191098+0000","last_fullsized":"2024-03-26T21:48:23.191098+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:15.544567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.544567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911447+0000","last_change":"2024-03-26T21:48:16.730412+0000","last_active":"2024-03-26T21:48:26.911447+0000","last_peered":"2024-03-26T21:48:26.911447+0000","last_clean":"2024-03-26T21:48:26.911447+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:48:26.911447+0000","last_undegraded":"2024-03-26T21:48:26.911447+0000","last_fullsized":"2024-03-26T21:48:26.911447+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.730303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.730303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:35.540578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":180,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919999+0000","last_change":"2024-03-26T21:44:49.392525+0000","last_active":"2024-03-26T21:48:21.919999+0000","last_peered":"2024-03-26T21:48:21.919999+0000","last_clean":"2024-03-26T21:48:21.919999+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:48:21.919999+0000","last_undegraded":"2024-03-26T21:48:21.919999+0000","last_fullsized":"2024-03-26T21:48:21.919999+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:59.627397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:43:59.627397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"448'3835","reported_seq":3875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.015935+0000","last_change":"2024-03-26T21:47:39.053277+0000","last_active":"2024-03-26T21:48:23.015935+0000","last_peered":"2024-03-26T21:48:23.015935+0000","last_clean":"2024-03-26T21:48:23.015935+0000","last_became_active":"2024-03-26T21:47:39.052942+0000","last_became_peered":"2024-03-26T21:47:39.052942+0000","last_unstale":"2024-03-26T21:48:23.015935+0000","last_undegraded":"2024-03-26T21:48:23.015935+0000","last_fullsized":"2024-03-26T21:48:23.015935+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.415225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251330560,"num_objects":3835,"num_object_clones":0,"num_object_copies":11505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":245440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"234'27","reported_seq":985,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.901354+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:48:21.901354+0000","last_peered":"2024-03-26T21:48:21.901354+0000","last_clean":"2024-03-26T21:48:21.901354+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:48:21.901354+0000","last_undegraded":"2024-03-26T21:48:21.901354+0000","last_fullsized":"2024-03-26T21:48:21.901354+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.150388+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.922170+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:48:21.922170+0000","last_peered":"2024-03-26T21:48:21.922170+0000","last_clean":"2024-03-26T21:48:21.922170+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:48:21.922170+0000","last_undegraded":"2024-03-26T21:48:21.922170+0000","last_fullsized":"2024-03-26T21:48:21.922170+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.420341+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":688,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:25.705118+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:48:25.705118+0000","last_peered":"2024-03-26T21:48:25.705118+0000","last_clean":"2024-03-26T21:48:25.705118+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:48:25.705118+0000","last_undegraded":"2024-03-26T21:48:25.705118+0000","last_fullsized":"2024-03-26T21:48:25.705118+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.930604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":835,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911339+0000","last_change":"2024-03-26T21:48:19.935968+0000","last_active":"2024-03-26T21:48:26.911339+0000","last_peered":"2024-03-26T21:48:26.911339+0000","last_clean":"2024-03-26T21:48:26.911339+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:48:26.911339+0000","last_undegraded":"2024-03-26T21:48:26.911339+0000","last_fullsized":"2024-03-26T21:48:26.911339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.935613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.935613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.386876+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"449'3745","reported_seq":3787,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.326139+0000","last_change":"2024-03-26T21:47:39.051954+0000","last_active":"2024-03-26T21:48:23.326139+0000","last_peered":"2024-03-26T21:48:23.326139+0000","last_clean":"2024-03-26T21:48:23.326139+0000","last_became_active":"2024-03-26T21:47:39.051498+0000","last_became_peered":"2024-03-26T21:47:39.051498+0000","last_unstale":"2024-03-26T21:48:23.326139+0000","last_undegraded":"2024-03-26T21:48:23.326139+0000","last_fullsized":"2024-03-26T21:48:23.326139+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3745,"log_dups_size":0,"ondisk_log_size":3745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:01.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245432320,"num_objects":3745,"num_object_clones":0,"num_object_copies":11235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3745,"num_write_kb":239680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":555,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191368+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:48:23.191368+0000","last_peered":"2024-03-26T21:48:23.191368+0000","last_clean":"2024-03-26T21:48:23.191368+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:48:23.191368+0000","last_undegraded":"2024-03-26T21:48:23.191368+0000","last_fullsized":"2024-03-26T21:48:23.191368+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.810891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"317'38","reported_seq":206,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921534+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:48:21.921534+0000","last_peered":"2024-03-26T21:48:21.921534+0000","last_clean":"2024-03-26T21:48:21.921534+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:48:21.921534+0000","last_undegraded":"2024-03-26T21:48:21.921534+0000","last_fullsized":"2024-03-26T21:48:21.921534+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:30.618938+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919821+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:48:21.919821+0000","last_peered":"2024-03-26T21:48:21.919821+0000","last_clean":"2024-03-26T21:48:21.919821+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:48:21.919821+0000","last_undegraded":"2024-03-26T21:48:21.919821+0000","last_fullsized":"2024-03-26T21:48:21.919821+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:18.541710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":548,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191941+0000","last_change":"2024-03-26T21:44:49.395042+0000","last_active":"2024-03-26T21:48:23.191941+0000","last_peered":"2024-03-26T21:48:23.191941+0000","last_clean":"2024-03-26T21:48:23.191941+0000","last_became_active":"2024-03-26T21:44:49.394613+0000","last_became_peered":"2024-03-26T21:44:49.394613+0000","last_unstale":"2024-03-26T21:48:23.191941+0000","last_undegraded":"2024-03-26T21:48:23.191941+0000","last_fullsized":"2024-03-26T21:48:23.191941+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:04.596247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:44:04.596247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"449'4022","reported_seq":4064,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.185531+0000","last_change":"2024-03-26T21:47:39.053658+0000","last_active":"2024-03-26T21:48:26.185531+0000","last_peered":"2024-03-26T21:48:26.185531+0000","last_clean":"2024-03-26T21:48:26.185531+0000","last_became_active":"2024-03-26T21:47:39.053244+0000","last_became_peered":"2024-03-26T21:47:39.053244+0000","last_unstale":"2024-03-26T21:48:26.185531+0000","last_undegraded":"2024-03-26T21:48:26.185531+0000","last_fullsized":"2024-03-26T21:48:26.185531+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4022,"log_dups_size":0,"ondisk_log_size":4022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:58.455414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263585792,"num_objects":4022,"num_object_clones":0,"num_object_copies":12066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":257408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":235,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.922265+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:48:21.922265+0000","last_peered":"2024-03-26T21:48:21.922265+0000","last_clean":"2024-03-26T21:48:21.922265+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:48:21.922265+0000","last_undegraded":"2024-03-26T21:48:21.922265+0000","last_fullsized":"2024-03-26T21:48:21.922265+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:27.516316+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":573,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191182+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:48:23.191182+0000","last_peered":"2024-03-26T21:48:23.191182+0000","last_clean":"2024-03-26T21:48:23.191182+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:48:23.191182+0000","last_undegraded":"2024-03-26T21:48:23.191182+0000","last_fullsized":"2024-03-26T21:48:23.191182+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:46.571871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":854,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181146+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:48:23.181146+0000","last_peered":"2024-03-26T21:48:23.181146+0000","last_clean":"2024-03-26T21:48:23.181146+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:48:23.181146+0000","last_undegraded":"2024-03-26T21:48:23.181146+0000","last_fullsized":"2024-03-26T21:48:23.181146+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.168894+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":759,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.182329+0000","last_change":"2024-03-26T21:48:11.992462+0000","last_active":"2024-03-26T21:48:23.182329+0000","last_peered":"2024-03-26T21:48:23.182329+0000","last_clean":"2024-03-26T21:48:23.182329+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:48:23.182329+0000","last_undegraded":"2024-03-26T21:48:23.182329+0000","last_fullsized":"2024-03-26T21:48:23.182329+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:31.860005+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"450'4052","reported_seq":4096,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:27.594417+0000","last_change":"2024-03-26T21:47:39.046470+0000","last_active":"2024-03-26T21:48:27.594417+0000","last_peered":"2024-03-26T21:48:27.594417+0000","last_clean":"2024-03-26T21:48:27.594417+0000","last_became_active":"2024-03-26T21:47:39.046152+0000","last_became_peered":"2024-03-26T21:47:39.046152+0000","last_unstale":"2024-03-26T21:48:27.594417+0000","last_undegraded":"2024-03-26T21:48:27.594417+0000","last_fullsized":"2024-03-26T21:48:27.594417+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:38.133188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4052,"num_write_kb":259328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1070,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179279+0000","last_change":"2024-03-26T21:46:11.496906+0000","last_active":"2024-03-26T21:48:23.179279+0000","last_peered":"2024-03-26T21:48:23.179279+0000","last_clean":"2024-03-26T21:48:23.179279+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:48:23.179279+0000","last_undegraded":"2024-03-26T21:48:23.179279+0000","last_fullsized":"2024-03-26T21:48:23.179279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:46:11.496758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:46:11.496758+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"234'20","reported_seq":917,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.900223+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:48:21.900223+0000","last_peered":"2024-03-26T21:48:21.900223+0000","last_clean":"2024-03-26T21:48:21.900223+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:48:21.900223+0000","last_undegraded":"2024-03-26T21:48:21.900223+0000","last_fullsized":"2024-03-26T21:48:21.900223+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:20.573732+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":860,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911619+0000","last_change":"2024-03-26T21:48:00.661315+0000","last_active":"2024-03-26T21:48:26.911619+0000","last_peered":"2024-03-26T21:48:26.911619+0000","last_clean":"2024-03-26T21:48:26.911619+0000","last_became_active":"2024-03-26T21:44:49.403379+0000","last_became_peered":"2024-03-26T21:44:49.403379+0000","last_unstale":"2024-03-26T21:48:26.911619+0000","last_undegraded":"2024-03-26T21:48:26.911619+0000","last_fullsized":"2024-03-26T21:48:26.911619+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:00.661167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:48:00.661167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:03.433855+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":832,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179942+0000","last_change":"2024-03-26T21:46:04.531935+0000","last_active":"2024-03-26T21:48:23.179942+0000","last_peered":"2024-03-26T21:48:23.179942+0000","last_clean":"2024-03-26T21:48:23.179942+0000","last_became_active":"2024-03-26T21:44:49.436816+0000","last_became_peered":"2024-03-26T21:44:49.436816+0000","last_unstale":"2024-03-26T21:48:23.179942+0000","last_undegraded":"2024-03-26T21:48:23.179942+0000","last_fullsized":"2024-03-26T21:48:23.179942+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:46:04.531770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:46:04.531770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"448'3781","reported_seq":3821,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:22.999595+0000","last_change":"2024-03-26T21:47:39.051081+0000","last_active":"2024-03-26T21:48:22.999595+0000","last_peered":"2024-03-26T21:48:22.999595+0000","last_clean":"2024-03-26T21:48:22.999595+0000","last_became_active":"2024-03-26T21:47:39.050784+0000","last_became_peered":"2024-03-26T21:47:39.050784+0000","last_unstale":"2024-03-26T21:48:22.999595+0000","last_undegraded":"2024-03-26T21:48:22.999595+0000","last_fullsized":"2024-03-26T21:48:22.999595+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.939822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":850,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911560+0000","last_change":"2024-03-26T21:48:02.087751+0000","last_active":"2024-03-26T21:48:26.911560+0000","last_peered":"2024-03-26T21:48:26.911560+0000","last_clean":"2024-03-26T21:48:26.911560+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:48:26.911560+0000","last_undegraded":"2024-03-26T21:48:26.911560+0000","last_fullsized":"2024-03-26T21:48:26.911560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.376172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":888,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179577+0000","last_change":"2024-03-26T21:47:16.673326+0000","last_active":"2024-03-26T21:48:23.179577+0000","last_peered":"2024-03-26T21:48:23.179577+0000","last_clean":"2024-03-26T21:48:23.179577+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:48:23.179577+0000","last_undegraded":"2024-03-26T21:48:23.179577+0000","last_fullsized":"2024-03-26T21:48:23.179577+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:47:16.673041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:16.673041+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":862,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.912125+0000","last_change":"2024-03-26T21:48:19.765008+0000","last_active":"2024-03-26T21:48:21.912125+0000","last_peered":"2024-03-26T21:48:21.912125+0000","last_clean":"2024-03-26T21:48:21.912125+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:48:21.912125+0000","last_undegraded":"2024-03-26T21:48:21.912125+0000","last_fullsized":"2024-03-26T21:48:21.912125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.881328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":834,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.902138+0000","last_change":"2024-03-26T21:47:15.451230+0000","last_active":"2024-03-26T21:48:21.902138+0000","last_peered":"2024-03-26T21:48:21.902138+0000","last_clean":"2024-03-26T21:48:21.902138+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:48:21.902138+0000","last_undegraded":"2024-03-26T21:48:21.902138+0000","last_fullsized":"2024-03-26T21:48:21.902138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:15.451119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.451119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:36.015426+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"448'3818","reported_seq":3858,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:22.967244+0000","last_change":"2024-03-26T21:47:39.047854+0000","last_active":"2024-03-26T21:48:22.967244+0000","last_peered":"2024-03-26T21:48:22.967244+0000","last_clean":"2024-03-26T21:48:22.967244+0000","last_became_active":"2024-03-26T21:47:39.047314+0000","last_became_peered":"2024-03-26T21:47:39.047314+0000","last_unstale":"2024-03-26T21:48:22.967244+0000","last_undegraded":"2024-03-26T21:48:22.967244+0000","last_fullsized":"2024-03-26T21:48:22.967244+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3818,"log_dups_size":0,"ondisk_log_size":3818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:07.969650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250216448,"num_objects":3818,"num_object_clones":0,"num_object_copies":11454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3818,"num_write_kb":244352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":840,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911536+0000","last_change":"2024-03-26T21:47:59.681466+0000","last_active":"2024-03-26T21:48:26.911536+0000","last_peered":"2024-03-26T21:48:26.911536+0000","last_clean":"2024-03-26T21:48:26.911536+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:48:26.911536+0000","last_undegraded":"2024-03-26T21:48:26.911536+0000","last_fullsized":"2024-03-26T21:48:26.911536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:59.681319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.681319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.002135+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":892,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181058+0000","last_change":"2024-03-26T21:48:11.911961+0000","last_active":"2024-03-26T21:48:23.181058+0000","last_peered":"2024-03-26T21:48:23.181058+0000","last_clean":"2024-03-26T21:48:23.181058+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:48:23.181058+0000","last_undegraded":"2024-03-26T21:48:23.181058+0000","last_fullsized":"2024-03-26T21:48:23.181058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:36.106001+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1881,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180739+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:48:23.180739+0000","last_peered":"2024-03-26T21:48:23.180739+0000","last_clean":"2024-03-26T21:48:23.180739+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:48:23.180739+0000","last_undegraded":"2024-03-26T21:48:23.180739+0000","last_fullsized":"2024-03-26T21:48:23.180739+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:38.910539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1067,"num_read_kb":1062,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":865,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.184526+0000","last_change":"2024-03-26T21:47:10.670076+0000","last_active":"2024-03-26T21:48:23.184526+0000","last_peered":"2024-03-26T21:48:23.184526+0000","last_clean":"2024-03-26T21:48:23.184526+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:48:23.184526+0000","last_undegraded":"2024-03-26T21:48:23.184526+0000","last_fullsized":"2024-03-26T21:48:23.184526+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:10.669965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:47:10.669965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:27.627217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":854,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.904017+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:48:21.904017+0000","last_peered":"2024-03-26T21:48:21.904017+0000","last_clean":"2024-03-26T21:48:21.904017+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:48:21.904017+0000","last_undegraded":"2024-03-26T21:48:21.904017+0000","last_fullsized":"2024-03-26T21:48:21.904017+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:38.368020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"448'3840","reported_seq":3880,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:22.948637+0000","last_change":"2024-03-26T21:47:39.047378+0000","last_active":"2024-03-26T21:48:22.948637+0000","last_peered":"2024-03-26T21:48:22.948637+0000","last_clean":"2024-03-26T21:48:22.948637+0000","last_became_active":"2024-03-26T21:47:39.047073+0000","last_became_peered":"2024-03-26T21:47:39.047073+0000","last_unstale":"2024-03-26T21:48:22.948637+0000","last_undegraded":"2024-03-26T21:48:22.948637+0000","last_fullsized":"2024-03-26T21:48:22.948637+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.356820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":3840,"num_object_clones":0,"num_object_copies":11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":776,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181208+0000","last_change":"2024-03-26T21:48:02.882565+0000","last_active":"2024-03-26T21:48:23.181208+0000","last_peered":"2024-03-26T21:48:23.181208+0000","last_clean":"2024-03-26T21:48:23.181208+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:48:23.181208+0000","last_undegraded":"2024-03-26T21:48:23.181208+0000","last_fullsized":"2024-03-26T21:48:23.181208+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.981858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":1212,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.913931+0000","last_change":"2024-03-26T21:48:23.427353+0000","last_active":"2024-03-26T21:48:26.913931+0000","last_peered":"2024-03-26T21:48:26.913931+0000","last_clean":"2024-03-26T21:48:26.913931+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:48:26.913931+0000","last_undegraded":"2024-03-26T21:48:26.913931+0000","last_fullsized":"2024-03-26T21:48:26.913931+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.771806+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921367+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:48:21.921367+0000","last_peered":"2024-03-26T21:48:21.921367+0000","last_clean":"2024-03-26T21:48:21.921367+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:48:21.921367+0000","last_undegraded":"2024-03-26T21:48:21.921367+0000","last_fullsized":"2024-03-26T21:48:21.921367+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:34.086717+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1036,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181186+0000","last_change":"2024-03-26T21:47:17.702466+0000","last_active":"2024-03-26T21:48:23.181186+0000","last_peered":"2024-03-26T21:48:23.181186+0000","last_clean":"2024-03-26T21:48:23.181186+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:48:23.181186+0000","last_undegraded":"2024-03-26T21:48:23.181186+0000","last_fullsized":"2024-03-26T21:48:23.181186+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:47:17.702328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:47:17.702328+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:24.868228+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911117+0000","last_change":"2024-03-26T21:48:05.886697+0000","last_active":"2024-03-26T21:48:26.911117+0000","last_peered":"2024-03-26T21:48:26.911117+0000","last_clean":"2024-03-26T21:48:26.911117+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:48:26.911117+0000","last_undegraded":"2024-03-26T21:48:26.911117+0000","last_fullsized":"2024-03-26T21:48:26.911117+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.886465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.886465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.802281+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"448'3861","reported_seq":3901,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:22.966654+0000","last_change":"2024-03-26T21:47:39.047146+0000","last_active":"2024-03-26T21:48:22.966654+0000","last_peered":"2024-03-26T21:48:22.966654+0000","last_clean":"2024-03-26T21:48:22.966654+0000","last_became_active":"2024-03-26T21:47:39.046837+0000","last_became_peered":"2024-03-26T21:47:39.046837+0000","last_unstale":"2024-03-26T21:48:22.966654+0000","last_undegraded":"2024-03-26T21:48:22.966654+0000","last_fullsized":"2024-03-26T21:48:22.966654+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.520342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":11583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":247104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.897845+0000","last_change":"2024-03-26T21:47:28.151192+0000","last_active":"2024-03-26T21:48:26.897845+0000","last_peered":"2024-03-26T21:48:26.897845+0000","last_clean":"2024-03-26T21:48:26.897845+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:48:26.897845+0000","last_undegraded":"2024-03-26T21:48:26.897845+0000","last_fullsized":"2024-03-26T21:48:26.897845+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.125869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.125869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:48.615066+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"234'22","reported_seq":888,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.901513+0000","last_change":"2024-03-26T21:47:19.374317+0000","last_active":"2024-03-26T21:48:21.901513+0000","last_peered":"2024-03-26T21:48:21.901513+0000","last_clean":"2024-03-26T21:48:21.901513+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:48:21.901513+0000","last_undegraded":"2024-03-26T21:48:21.901513+0000","last_fullsized":"2024-03-26T21:48:21.901513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:47:19.374222+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:47:19.374222+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:30.550904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":885,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896941+0000","last_change":"2024-03-26T21:48:17.180103+0000","last_active":"2024-03-26T21:48:26.896941+0000","last_peered":"2024-03-26T21:48:26.896941+0000","last_clean":"2024-03-26T21:48:26.896941+0000","last_became_active":"2024-03-26T21:44:49.410374+0000","last_became_peered":"2024-03-26T21:44:49.410374+0000","last_unstale":"2024-03-26T21:48:26.896941+0000","last_undegraded":"2024-03-26T21:48:26.896941+0000","last_fullsized":"2024-03-26T21:48:26.896941+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.339582+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":862,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.901093+0000","last_change":"2024-03-26T21:48:05.696135+0000","last_active":"2024-03-26T21:48:21.901093+0000","last_peered":"2024-03-26T21:48:21.901093+0000","last_clean":"2024-03-26T21:48:21.901093+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:48:21.901093+0000","last_undegraded":"2024-03-26T21:48:21.901093+0000","last_fullsized":"2024-03-26T21:48:21.901093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:30.580793+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":756,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181203+0000","last_change":"2024-03-26T21:47:19.718976+0000","last_active":"2024-03-26T21:48:23.181203+0000","last_peered":"2024-03-26T21:48:23.181203+0000","last_clean":"2024-03-26T21:48:23.181203+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:48:23.181203+0000","last_undegraded":"2024-03-26T21:48:23.181203+0000","last_fullsized":"2024-03-26T21:48:23.181203+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:47:19.718867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:30.854147+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"448'3742","reported_seq":3781,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:22.972435+0000","last_change":"2024-03-26T21:47:39.051677+0000","last_active":"2024-03-26T21:48:22.972435+0000","last_peered":"2024-03-26T21:48:22.972435+0000","last_clean":"2024-03-26T21:48:22.972435+0000","last_became_active":"2024-03-26T21:47:39.051279+0000","last_became_peered":"2024-03-26T21:47:39.051279+0000","last_unstale":"2024-03-26T21:48:22.972435+0000","last_undegraded":"2024-03-26T21:48:22.972435+0000","last_fullsized":"2024-03-26T21:48:22.972435+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3742,"log_dups_size":0,"ondisk_log_size":3742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:39.668253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245235712,"num_objects":3742,"num_object_clones":0,"num_object_copies":11226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":239488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":845,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.178810+0000","last_change":"2024-03-26T21:46:19.552900+0000","last_active":"2024-03-26T21:48:23.178810+0000","last_peered":"2024-03-26T21:48:23.178810+0000","last_clean":"2024-03-26T21:48:23.178810+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:48:23.178810+0000","last_undegraded":"2024-03-26T21:48:23.178810+0000","last_fullsized":"2024-03-26T21:48:23.178810+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:46:19.552767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:46:19.552767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"364'43","reported_seq":1160,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910239+0000","last_change":"2024-03-26T21:48:14.657443+0000","last_active":"2024-03-26T21:48:26.910239+0000","last_peered":"2024-03-26T21:48:26.910239+0000","last_clean":"2024-03-26T21:48:26.910239+0000","last_became_active":"2024-03-26T21:44:49.397229+0000","last_became_peered":"2024-03-26T21:44:49.397229+0000","last_unstale":"2024-03-26T21:48:26.910239+0000","last_undegraded":"2024-03-26T21:48:26.910239+0000","last_fullsized":"2024-03-26T21:48:26.910239+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'43","last_scrub_stamp":"2024-03-26T21:48:14.657283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.657283+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.781065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896709+0000","last_change":"2024-03-26T21:48:14.161052+0000","last_active":"2024-03-26T21:48:26.896709+0000","last_peered":"2024-03-26T21:48:26.896709+0000","last_clean":"2024-03-26T21:48:26.896709+0000","last_became_active":"2024-03-26T21:43:16.796261+0000","last_became_peered":"2024-03-26T21:43:16.796261+0000","last_unstale":"2024-03-26T21:48:26.896709+0000","last_undegraded":"2024-03-26T21:48:26.896709+0000","last_fullsized":"2024-03-26T21:48:26.896709+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.059211+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":858,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910899+0000","last_change":"2024-03-26T21:47:12.568606+0000","last_active":"2024-03-26T21:48:26.910899+0000","last_peered":"2024-03-26T21:48:26.910899+0000","last_clean":"2024-03-26T21:48:26.910899+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:48:26.910899+0000","last_undegraded":"2024-03-26T21:48:26.910899+0000","last_fullsized":"2024-03-26T21:48:26.910899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:47:12.568413+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:47:12.568413+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.568413+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:30.352770+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":180,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919193+0000","last_change":"2024-03-26T21:44:49.402539+0000","last_active":"2024-03-26T21:48:21.919193+0000","last_peered":"2024-03-26T21:48:21.919193+0000","last_clean":"2024-03-26T21:48:21.919193+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:48:21.919193+0000","last_undegraded":"2024-03-26T21:48:21.919193+0000","last_fullsized":"2024-03-26T21:48:21.919193+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:43:56.468289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:43:56.468289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"449'3853","reported_seq":3895,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.321162+0000","last_change":"2024-03-26T21:47:39.048784+0000","last_active":"2024-03-26T21:48:23.321162+0000","last_peered":"2024-03-26T21:48:23.321162+0000","last_clean":"2024-03-26T21:48:23.321162+0000","last_became_active":"2024-03-26T21:47:39.047884+0000","last_became_peered":"2024-03-26T21:47:39.047884+0000","last_unstale":"2024-03-26T21:48:23.321162+0000","last_undegraded":"2024-03-26T21:48:23.321162+0000","last_fullsized":"2024-03-26T21:48:23.321162+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252510208,"num_objects":3853,"num_object_clones":0,"num_object_copies":11559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":246592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":885,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179725+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:48:23.179725+0000","last_peered":"2024-03-26T21:48:23.179725+0000","last_clean":"2024-03-26T21:48:23.179725+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:48:23.179725+0000","last_undegraded":"2024-03-26T21:48:23.179725+0000","last_fullsized":"2024-03-26T21:48:23.179725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:17.027797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":867,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910956+0000","last_change":"2024-03-26T21:48:23.858924+0000","last_active":"2024-03-26T21:48:26.910956+0000","last_peered":"2024-03-26T21:48:26.910956+0000","last_clean":"2024-03-26T21:48:26.910956+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:48:26.910956+0000","last_undegraded":"2024-03-26T21:48:26.910956+0000","last_fullsized":"2024-03-26T21:48:26.910956+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.858784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.858784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:25.054542+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":883,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896759+0000","last_change":"2024-03-26T21:48:04.064240+0000","last_active":"2024-03-26T21:48:26.896759+0000","last_peered":"2024-03-26T21:48:26.896759+0000","last_clean":"2024-03-26T21:48:26.896759+0000","last_became_active":"2024-03-26T21:44:49.410172+0000","last_became_peered":"2024-03-26T21:44:49.410172+0000","last_unstale":"2024-03-26T21:48:26.896759+0000","last_undegraded":"2024-03-26T21:48:26.896759+0000","last_fullsized":"2024-03-26T21:48:26.896759+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:09.204031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":829,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.184265+0000","last_change":"2024-03-26T21:48:03.013891+0000","last_active":"2024-03-26T21:48:23.184265+0000","last_peered":"2024-03-26T21:48:23.184265+0000","last_clean":"2024-03-26T21:48:23.184265+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:48:23.184265+0000","last_undegraded":"2024-03-26T21:48:23.184265+0000","last_fullsized":"2024-03-26T21:48:23.184265+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.674529+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"449'3644","reported_seq":3686,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.333054+0000","last_change":"2024-03-26T21:47:39.046775+0000","last_active":"2024-03-26T21:48:23.333054+0000","last_peered":"2024-03-26T21:48:23.333054+0000","last_clean":"2024-03-26T21:48:23.333054+0000","last_became_active":"2024-03-26T21:47:39.046369+0000","last_became_peered":"2024-03-26T21:47:39.046369+0000","last_unstale":"2024-03-26T21:48:23.333054+0000","last_undegraded":"2024-03-26T21:48:23.333054+0000","last_fullsized":"2024-03-26T21:48:23.333054+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:06.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3644,"num_write_kb":233216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":927,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911447+0000","last_change":"2024-03-26T21:48:01.054496+0000","last_active":"2024-03-26T21:48:26.911447+0000","last_peered":"2024-03-26T21:48:26.911447+0000","last_clean":"2024-03-26T21:48:26.911447+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:48:26.911447+0000","last_undegraded":"2024-03-26T21:48:26.911447+0000","last_fullsized":"2024-03-26T21:48:26.911447+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.422288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":791,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181721+0000","last_change":"2024-03-26T21:47:28.835918+0000","last_active":"2024-03-26T21:48:23.181721+0000","last_peered":"2024-03-26T21:48:23.181721+0000","last_clean":"2024-03-26T21:48:23.181721+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:48:23.181721+0000","last_undegraded":"2024-03-26T21:48:23.181721+0000","last_fullsized":"2024-03-26T21:48:23.181721+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.835830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.835830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:42.287565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896934+0000","last_change":"2024-03-26T21:48:02.055651+0000","last_active":"2024-03-26T21:48:26.896934+0000","last_peered":"2024-03-26T21:48:26.896934+0000","last_clean":"2024-03-26T21:48:26.896934+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:48:26.896934+0000","last_undegraded":"2024-03-26T21:48:26.896934+0000","last_fullsized":"2024-03-26T21:48:26.896934+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:10.758127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":764,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180736+0000","last_change":"2024-03-26T21:48:10.897127+0000","last_active":"2024-03-26T21:48:23.180736+0000","last_peered":"2024-03-26T21:48:23.180736+0000","last_clean":"2024-03-26T21:48:23.180736+0000","last_became_active":"2024-03-26T21:44:49.396166+0000","last_became_peered":"2024-03-26T21:44:49.396166+0000","last_unstale":"2024-03-26T21:48:23.180736+0000","last_undegraded":"2024-03-26T21:48:23.180736+0000","last_fullsized":"2024-03-26T21:48:23.180736+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:19.451684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"449'3786","reported_seq":3828,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.977368+0000","last_change":"2024-03-26T21:47:39.055017+0000","last_active":"2024-03-26T21:48:23.977368+0000","last_peered":"2024-03-26T21:48:23.977368+0000","last_clean":"2024-03-26T21:48:23.977368+0000","last_became_active":"2024-03-26T21:47:39.054705+0000","last_became_peered":"2024-03-26T21:47:39.054705+0000","last_unstale":"2024-03-26T21:48:23.977368+0000","last_undegraded":"2024-03-26T21:48:23.977368+0000","last_fullsized":"2024-03-26T21:48:23.977368+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:58.901128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":11358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":242304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":866,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.896710+0000","last_change":"2024-03-26T21:48:23.129601+0000","last_active":"2024-03-26T21:48:26.896710+0000","last_peered":"2024-03-26T21:48:26.896710+0000","last_clean":"2024-03-26T21:48:26.896710+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:48:26.896710+0000","last_undegraded":"2024-03-26T21:48:26.896710+0000","last_fullsized":"2024-03-26T21:48:26.896710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.123676+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":590,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191270+0000","last_change":"2024-03-26T21:44:49.491836+0000","last_active":"2024-03-26T21:48:23.191270+0000","last_peered":"2024-03-26T21:48:23.191270+0000","last_clean":"2024-03-26T21:48:23.191270+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:48:23.191270+0000","last_undegraded":"2024-03-26T21:48:23.191270+0000","last_fullsized":"2024-03-26T21:48:23.191270+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:44:00.448307+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:41:43.970479+0000","last_clean_scrub_stamp":"2024-03-26T21:44:00.448307+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":821,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181527+0000","last_change":"2024-03-26T21:47:22.749199+0000","last_active":"2024-03-26T21:48:23.181527+0000","last_peered":"2024-03-26T21:48:23.181527+0000","last_clean":"2024-03-26T21:48:23.181527+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:48:23.181527+0000","last_undegraded":"2024-03-26T21:48:23.181527+0000","last_fullsized":"2024-03-26T21:48:23.181527+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:47:22.749114+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:29:44.612711+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.749114+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:34.803570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921692+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:48:21.921692+0000","last_peered":"2024-03-26T21:48:21.921692+0000","last_clean":"2024-03-26T21:48:21.921692+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:48:21.921692+0000","last_undegraded":"2024-03-26T21:48:21.921692+0000","last_fullsized":"2024-03-26T21:48:21.921692+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.322569+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"450'4142","reported_seq":4186,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:27.098397+0000","last_change":"2024-03-26T21:47:39.051767+0000","last_active":"2024-03-26T21:48:27.098397+0000","last_peered":"2024-03-26T21:48:27.098397+0000","last_clean":"2024-03-26T21:48:27.098397+0000","last_became_active":"2024-03-26T21:47:39.051349+0000","last_became_peered":"2024-03-26T21:47:39.051349+0000","last_unstale":"2024-03-26T21:48:27.098397+0000","last_undegraded":"2024-03-26T21:48:27.098397+0000","last_fullsized":"2024-03-26T21:48:27.098397+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:55.642445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271450112,"num_objects":4142,"num_object_clones":0,"num_object_copies":12426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4142,"num_write_kb":265088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1022,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180855+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:48:23.180855+0000","last_peered":"2024-03-26T21:48:23.180855+0000","last_clean":"2024-03-26T21:48:23.180855+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:48:23.180855+0000","last_undegraded":"2024-03-26T21:48:23.180855+0000","last_fullsized":"2024-03-26T21:48:23.180855+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.821327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":896,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910590+0000","last_change":"2024-03-26T21:48:17.779307+0000","last_active":"2024-03-26T21:48:26.910590+0000","last_peered":"2024-03-26T21:48:26.910590+0000","last_clean":"2024-03-26T21:48:26.910590+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:48:26.910590+0000","last_undegraded":"2024-03-26T21:48:26.910590+0000","last_fullsized":"2024-03-26T21:48:26.910590+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:17.779045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.779045+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:33.002288+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":838,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181561+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:48:23.181561+0000","last_peered":"2024-03-26T21:48:23.181561+0000","last_clean":"2024-03-26T21:48:23.181561+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:48:23.181561+0000","last_undegraded":"2024-03-26T21:48:23.181561+0000","last_fullsized":"2024-03-26T21:48:23.181561+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:31.675781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":836,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:27.000688+0000","last_change":"2024-03-26T21:48:27.000729+0000","last_active":"2024-03-26T21:48:27.000688+0000","last_peered":"2024-03-26T21:48:27.000688+0000","last_clean":"2024-03-26T21:48:27.000688+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:48:27.000688+0000","last_undegraded":"2024-03-26T21:48:27.000688+0000","last_fullsized":"2024-03-26T21:48:27.000688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:27.000615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.000615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.371457+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"234'12","reported_seq":196,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919495+0000","last_change":"2024-03-26T21:47:59.714612+0000","last_active":"2024-03-26T21:48:21.919495+0000","last_peered":"2024-03-26T21:48:21.919495+0000","last_clean":"2024-03-26T21:48:21.919495+0000","last_became_active":"2024-03-26T21:44:49.408457+0000","last_became_peered":"2024-03-26T21:44:49.408457+0000","last_unstale":"2024-03-26T21:48:21.919495+0000","last_undegraded":"2024-03-26T21:48:21.919495+0000","last_fullsized":"2024-03-26T21:48:21.919495+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:06.709447+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.907865+0000","last_change":"2024-03-26T21:47:26.982275+0000","last_active":"2024-03-26T21:48:26.907865+0000","last_peered":"2024-03-26T21:48:26.907865+0000","last_clean":"2024-03-26T21:48:26.907865+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:48:26.907865+0000","last_undegraded":"2024-03-26T21:48:26.907865+0000","last_fullsized":"2024-03-26T21:48:26.907865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:26.982173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.982173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:36.849597+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":851,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911101+0000","last_change":"2024-03-26T21:48:07.714924+0000","last_active":"2024-03-26T21:48:26.911101+0000","last_peered":"2024-03-26T21:48:26.911101+0000","last_clean":"2024-03-26T21:48:26.911101+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:48:26.911101+0000","last_undegraded":"2024-03-26T21:48:26.911101+0000","last_fullsized":"2024-03-26T21:48:26.911101+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.714744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.714744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:33.078685+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":773,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181345+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:48:23.181345+0000","last_peered":"2024-03-26T21:48:23.181345+0000","last_clean":"2024-03-26T21:48:23.181345+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:48:23.181345+0000","last_undegraded":"2024-03-26T21:48:23.181345+0000","last_fullsized":"2024-03-26T21:48:23.181345+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:38.657067+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":895,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181809+0000","last_change":"2024-03-26T21:47:11.703126+0000","last_active":"2024-03-26T21:48:23.181809+0000","last_peered":"2024-03-26T21:48:23.181809+0000","last_clean":"2024-03-26T21:48:23.181809+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:48:23.181809+0000","last_undegraded":"2024-03-26T21:48:23.181809+0000","last_fullsized":"2024-03-26T21:48:23.181809+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:47:11.703039+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.703039+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:29.276097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":811,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181351+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:48:23.181351+0000","last_peered":"2024-03-26T21:48:23.181351+0000","last_clean":"2024-03-26T21:48:23.181351+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:48:23.181351+0000","last_undegraded":"2024-03-26T21:48:23.181351+0000","last_fullsized":"2024-03-26T21:48:23.181351+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:11.797721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919671+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:48:21.919671+0000","last_peered":"2024-03-26T21:48:21.919671+0000","last_clean":"2024-03-26T21:48:21.919671+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:48:21.919671+0000","last_undegraded":"2024-03-26T21:48:21.919671+0000","last_fullsized":"2024-03-26T21:48:21.919671+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.275926+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":867,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.897302+0000","last_change":"2024-03-26T21:47:18.904009+0000","last_active":"2024-03-26T21:48:26.897302+0000","last_peered":"2024-03-26T21:48:26.897302+0000","last_clean":"2024-03-26T21:48:26.897302+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:48:26.897302+0000","last_undegraded":"2024-03-26T21:48:26.897302+0000","last_fullsized":"2024-03-26T21:48:26.897302+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:45.184979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":911,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911819+0000","last_change":"2024-03-26T21:48:20.776635+0000","last_active":"2024-03-26T21:48:26.911819+0000","last_peered":"2024-03-26T21:48:26.911819+0000","last_clean":"2024-03-26T21:48:26.911819+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:48:26.911819+0000","last_undegraded":"2024-03-26T21:48:26.911819+0000","last_fullsized":"2024-03-26T21:48:26.911819+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:20.776494+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.776494+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.363911+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":850,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179711+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:48:23.179711+0000","last_peered":"2024-03-26T21:48:23.179711+0000","last_clean":"2024-03-26T21:48:23.179711+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:48:23.179711+0000","last_undegraded":"2024-03-26T21:48:23.179711+0000","last_fullsized":"2024-03-26T21:48:23.179711+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:37.946686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"364'13","reported_seq":227,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.920236+0000","last_change":"2024-03-26T21:44:49.448360+0000","last_active":"2024-03-26T21:48:21.920236+0000","last_peered":"2024-03-26T21:48:21.920236+0000","last_clean":"2024-03-26T21:48:21.920236+0000","last_became_active":"2024-03-26T21:44:49.402177+0000","last_became_peered":"2024-03-26T21:44:49.402177+0000","last_unstale":"2024-03-26T21:48:21.920236+0000","last_undegraded":"2024-03-26T21:48:21.920236+0000","last_fullsized":"2024-03-26T21:48:21.920236+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'12","last_scrub_stamp":"2024-03-26T21:44:06.340020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:44:06.340020+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":534,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.194864+0000","last_change":"2024-03-26T21:47:58.705838+0000","last_active":"2024-03-26T21:48:23.194864+0000","last_peered":"2024-03-26T21:48:23.194864+0000","last_clean":"2024-03-26T21:48:23.194864+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:48:23.194864+0000","last_undegraded":"2024-03-26T21:48:23.194864+0000","last_fullsized":"2024-03-26T21:48:23.194864+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:00.410661+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"234'25","reported_seq":200,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919891+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:48:21.919891+0000","last_peered":"2024-03-26T21:48:21.919891+0000","last_clean":"2024-03-26T21:48:21.919891+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:48:21.919891+0000","last_undegraded":"2024-03-26T21:48:21.919891+0000","last_fullsized":"2024-03-26T21:48:21.919891+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:19.599750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":784,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.182440+0000","last_change":"2024-03-26T21:48:08.932035+0000","last_active":"2024-03-26T21:48:23.182440+0000","last_peered":"2024-03-26T21:48:23.182440+0000","last_clean":"2024-03-26T21:48:23.182440+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:48:23.182440+0000","last_undegraded":"2024-03-26T21:48:23.182440+0000","last_fullsized":"2024-03-26T21:48:23.182440+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:10.831931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1079,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911523+0000","last_change":"2024-03-26T21:47:05.572410+0000","last_active":"2024-03-26T21:48:26.911523+0000","last_peered":"2024-03-26T21:48:26.911523+0000","last_clean":"2024-03-26T21:48:26.911523+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:48:26.911523+0000","last_undegraded":"2024-03-26T21:48:26.911523+0000","last_fullsized":"2024-03-26T21:48:26.911523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:47:05.572334+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:47:05.572334+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:29.337876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":847,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180961+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:48:23.180961+0000","last_peered":"2024-03-26T21:48:23.180961+0000","last_clean":"2024-03-26T21:48:23.180961+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:48:23.180961+0000","last_undegraded":"2024-03-26T21:48:23.180961+0000","last_fullsized":"2024-03-26T21:48:23.180961+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:46.622715+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"424'24","reported_seq":564,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.192286+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:48:23.192286+0000","last_peered":"2024-03-26T21:48:23.192286+0000","last_clean":"2024-03-26T21:48:23.192286+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:48:23.192286+0000","last_undegraded":"2024-03-26T21:48:23.192286+0000","last_fullsized":"2024-03-26T21:48:23.192286+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:34.977736+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":536,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.193274+0000","last_change":"2024-03-26T21:47:04.478046+0000","last_active":"2024-03-26T21:48:23.193274+0000","last_peered":"2024-03-26T21:48:23.193274+0000","last_clean":"2024-03-26T21:48:23.193274+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:48:23.193274+0000","last_undegraded":"2024-03-26T21:48:23.193274+0000","last_fullsized":"2024-03-26T21:48:23.193274+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:04.477962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:04.477962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":861,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.900110+0000","last_change":"2024-03-26T21:47:05.535987+0000","last_active":"2024-03-26T21:48:21.900110+0000","last_peered":"2024-03-26T21:48:21.900110+0000","last_clean":"2024-03-26T21:48:21.900110+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:48:21.900110+0000","last_undegraded":"2024-03-26T21:48:21.900110+0000","last_fullsized":"2024-03-26T21:48:21.900110+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:05.535894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:05.535894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:24.669654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919087+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:48:21.919087+0000","last_peered":"2024-03-26T21:48:21.919087+0000","last_clean":"2024-03-26T21:48:21.919087+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:48:21.919087+0000","last_undegraded":"2024-03-26T21:48:21.919087+0000","last_fullsized":"2024-03-26T21:48:21.919087+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:29.838067+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":870,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.900326+0000","last_change":"2024-03-26T21:47:11.457655+0000","last_active":"2024-03-26T21:48:21.900326+0000","last_peered":"2024-03-26T21:48:21.900326+0000","last_clean":"2024-03-26T21:48:21.900326+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:48:21.900326+0000","last_undegraded":"2024-03-26T21:48:21.900326+0000","last_fullsized":"2024-03-26T21:48:21.900326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:47:11.457513+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.457513+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:29.849940+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":788,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181455+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:48:23.181455+0000","last_peered":"2024-03-26T21:48:23.181455+0000","last_clean":"2024-03-26T21:48:23.181455+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:48:23.181455+0000","last_undegraded":"2024-03-26T21:48:23.181455+0000","last_fullsized":"2024-03-26T21:48:23.181455+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.032406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921468+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:48:21.921468+0000","last_peered":"2024-03-26T21:48:21.921468+0000","last_clean":"2024-03-26T21:48:21.921468+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:48:21.921468+0000","last_undegraded":"2024-03-26T21:48:21.921468+0000","last_fullsized":"2024-03-26T21:48:21.921468+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.201839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":827,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179657+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:48:23.179657+0000","last_peered":"2024-03-26T21:48:23.179657+0000","last_clean":"2024-03-26T21:48:23.179657+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:48:23.179657+0000","last_undegraded":"2024-03-26T21:48:23.179657+0000","last_fullsized":"2024-03-26T21:48:23.179657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:13.726112+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":200,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919452+0000","last_change":"2024-03-26T21:47:08.400219+0000","last_active":"2024-03-26T21:48:21.919452+0000","last_peered":"2024-03-26T21:48:21.919452+0000","last_clean":"2024-03-26T21:48:21.919452+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:48:21.919452+0000","last_undegraded":"2024-03-26T21:48:21.919452+0000","last_fullsized":"2024-03-26T21:48:21.919452+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:47:08.400013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:08.400013+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":856,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910823+0000","last_change":"2024-03-26T21:48:12.733553+0000","last_active":"2024-03-26T21:48:26.910823+0000","last_peered":"2024-03-26T21:48:26.910823+0000","last_clean":"2024-03-26T21:48:26.910823+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:48:26.910823+0000","last_undegraded":"2024-03-26T21:48:26.910823+0000","last_fullsized":"2024-03-26T21:48:26.910823+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.733428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.733428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.412196+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":766,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181132+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:48:23.181132+0000","last_peered":"2024-03-26T21:48:23.181132+0000","last_clean":"2024-03-26T21:48:23.181132+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:48:23.181132+0000","last_undegraded":"2024-03-26T21:48:23.181132+0000","last_fullsized":"2024-03-26T21:48:23.181132+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.855255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921576+0000","last_change":"2024-03-26T21:47:07.534779+0000","last_active":"2024-03-26T21:48:21.921576+0000","last_peered":"2024-03-26T21:48:21.921576+0000","last_clean":"2024-03-26T21:48:21.921576+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:48:21.921576+0000","last_undegraded":"2024-03-26T21:48:21.921576+0000","last_fullsized":"2024-03-26T21:48:21.921576+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:07.534655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:47:07.534655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":194,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921959+0000","last_change":"2024-03-26T21:47:16.302095+0000","last_active":"2024-03-26T21:48:21.921959+0000","last_peered":"2024-03-26T21:48:21.921959+0000","last_clean":"2024-03-26T21:48:21.921959+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:48:21.921959+0000","last_undegraded":"2024-03-26T21:48:21.921959+0000","last_fullsized":"2024-03-26T21:48:21.921959+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:47:16.301826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:16.301826+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:24.067358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":851,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179116+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:48:23.179116+0000","last_peered":"2024-03-26T21:48:23.179116+0000","last_clean":"2024-03-26T21:48:23.179116+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:48:23.179116+0000","last_undegraded":"2024-03-26T21:48:23.179116+0000","last_fullsized":"2024-03-26T21:48:23.179116+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:26.599241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":879,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911980+0000","last_change":"2024-03-26T21:48:26.476675+0000","last_active":"2024-03-26T21:48:26.911980+0000","last_peered":"2024-03-26T21:48:26.911980+0000","last_clean":"2024-03-26T21:48:26.911980+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:48:26.911980+0000","last_undegraded":"2024-03-26T21:48:26.911980+0000","last_fullsized":"2024-03-26T21:48:26.911980+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.591969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":774,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180899+0000","last_change":"2024-03-26T21:48:22.860635+0000","last_active":"2024-03-26T21:48:23.180899+0000","last_peered":"2024-03-26T21:48:23.180899+0000","last_clean":"2024-03-26T21:48:23.180899+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:48:23.180899+0000","last_undegraded":"2024-03-26T21:48:23.180899+0000","last_fullsized":"2024-03-26T21:48:23.180899+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.867613+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":890,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911917+0000","last_change":"2024-03-26T21:47:05.766243+0000","last_active":"2024-03-26T21:48:26.911917+0000","last_peered":"2024-03-26T21:48:26.911917+0000","last_clean":"2024-03-26T21:48:26.911917+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:48:26.911917+0000","last_undegraded":"2024-03-26T21:48:26.911917+0000","last_fullsized":"2024-03-26T21:48:26.911917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:47:05.766149+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:47:05.766149+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":180,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.921427+0000","last_change":"2024-03-26T21:44:49.426244+0000","last_active":"2024-03-26T21:48:21.921427+0000","last_peered":"2024-03-26T21:48:21.921427+0000","last_clean":"2024-03-26T21:48:21.921427+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:48:21.921427+0000","last_undegraded":"2024-03-26T21:48:21.921427+0000","last_fullsized":"2024-03-26T21:48:21.921427+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:44:05.251423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:44:05.251423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":833,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181962+0000","last_change":"2024-03-26T21:47:22.710800+0000","last_active":"2024-03-26T21:48:23.181962+0000","last_peered":"2024-03-26T21:48:23.181962+0000","last_clean":"2024-03-26T21:48:23.181962+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:48:23.181962+0000","last_undegraded":"2024-03-26T21:48:23.181962+0000","last_fullsized":"2024-03-26T21:48:23.181962+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:22.710723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.710723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:29.662404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":861,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.018018+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:48:23.018018+0000","last_peered":"2024-03-26T21:48:23.018018+0000","last_clean":"2024-03-26T21:48:23.018018+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:48:23.018018+0000","last_undegraded":"2024-03-26T21:48:23.018018+0000","last_fullsized":"2024-03-26T21:48:23.018018+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.137319+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":919,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181699+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:48:23.181699+0000","last_peered":"2024-03-26T21:48:23.181699+0000","last_clean":"2024-03-26T21:48:23.181699+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:48:23.181699+0000","last_undegraded":"2024-03-26T21:48:23.181699+0000","last_fullsized":"2024-03-26T21:48:23.181699+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.500276+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":896,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.903217+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:48:21.903217+0000","last_peered":"2024-03-26T21:48:21.903217+0000","last_clean":"2024-03-26T21:48:21.903217+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:48:21.903217+0000","last_undegraded":"2024-03-26T21:48:21.903217+0000","last_fullsized":"2024-03-26T21:48:21.903217+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.571019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":1135,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.913315+0000","last_change":"2024-03-26T21:47:22.969388+0000","last_active":"2024-03-26T21:48:26.913315+0000","last_peered":"2024-03-26T21:48:26.913315+0000","last_clean":"2024-03-26T21:48:26.913315+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:48:26.913315+0000","last_undegraded":"2024-03-26T21:48:26.913315+0000","last_fullsized":"2024-03-26T21:48:26.913315+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:22.969296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.969296+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":769,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.181089+0000","last_change":"2024-03-26T21:48:04.881122+0000","last_active":"2024-03-26T21:48:23.181089+0000","last_peered":"2024-03-26T21:48:23.181089+0000","last_clean":"2024-03-26T21:48:23.181089+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:48:23.181089+0000","last_undegraded":"2024-03-26T21:48:23.181089+0000","last_fullsized":"2024-03-26T21:48:23.181089+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:12.034875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'23","reported_seq":205,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.920929+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:48:21.920929+0000","last_peered":"2024-03-26T21:48:21.920929+0000","last_clean":"2024-03-26T21:48:21.920929+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:48:21.920929+0000","last_undegraded":"2024-03-26T21:48:21.920929+0000","last_fullsized":"2024-03-26T21:48:21.920929+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:26.918773+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":535,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191750+0000","last_change":"2024-03-26T21:48:07.663645+0000","last_active":"2024-03-26T21:48:23.191750+0000","last_peered":"2024-03-26T21:48:23.191750+0000","last_clean":"2024-03-26T21:48:23.191750+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:48:23.191750+0000","last_undegraded":"2024-03-26T21:48:23.191750+0000","last_fullsized":"2024-03-26T21:48:23.191750+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:37.361646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":559,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.193465+0000","last_change":"2024-03-26T21:48:16.766872+0000","last_active":"2024-03-26T21:48:23.193465+0000","last_peered":"2024-03-26T21:48:23.193465+0000","last_clean":"2024-03-26T21:48:23.193465+0000","last_became_active":"2024-03-26T21:44:49.395939+0000","last_became_peered":"2024-03-26T21:44:49.395939+0000","last_unstale":"2024-03-26T21:48:23.193465+0000","last_undegraded":"2024-03-26T21:48:23.193465+0000","last_fullsized":"2024-03-26T21:48:23.193465+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.981952+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919716+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:48:21.919716+0000","last_peered":"2024-03-26T21:48:21.919716+0000","last_clean":"2024-03-26T21:48:21.919716+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:48:21.919716+0000","last_undegraded":"2024-03-26T21:48:21.919716+0000","last_fullsized":"2024-03-26T21:48:21.919716+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:26.679721+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":947,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.897016+0000","last_change":"2024-03-26T21:48:09.176526+0000","last_active":"2024-03-26T21:48:26.897016+0000","last_peered":"2024-03-26T21:48:26.897016+0000","last_clean":"2024-03-26T21:48:26.897016+0000","last_became_active":"2024-03-26T21:44:49.417143+0000","last_became_peered":"2024-03-26T21:44:49.417143+0000","last_unstale":"2024-03-26T21:48:26.897016+0000","last_undegraded":"2024-03-26T21:48:26.897016+0000","last_fullsized":"2024-03-26T21:48:26.897016+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:25.793568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":852,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.179485+0000","last_change":"2024-03-26T21:47:12.688818+0000","last_active":"2024-03-26T21:48:23.179485+0000","last_peered":"2024-03-26T21:48:23.179485+0000","last_clean":"2024-03-26T21:48:23.179485+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:48:23.179485+0000","last_undegraded":"2024-03-26T21:48:23.179485+0000","last_fullsized":"2024-03-26T21:48:23.179485+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.688726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.688726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":857,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.900071+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:48:21.900071+0000","last_peered":"2024-03-26T21:48:21.900071+0000","last_clean":"2024-03-26T21:48:21.900071+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:48:21.900071+0000","last_undegraded":"2024-03-26T21:48:21.900071+0000","last_fullsized":"2024-03-26T21:48:21.900071+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.449043+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":536,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.194739+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:48:23.194739+0000","last_peered":"2024-03-26T21:48:23.194739+0000","last_clean":"2024-03-26T21:48:23.194739+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:48:23.194739+0000","last_undegraded":"2024-03-26T21:48:23.194739+0000","last_fullsized":"2024-03-26T21:48:23.194739+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:43.666911+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":568,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.192320+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:48:23.192320+0000","last_peered":"2024-03-26T21:48:23.192320+0000","last_clean":"2024-03-26T21:48:23.192320+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:48:23.192320+0000","last_undegraded":"2024-03-26T21:48:23.192320+0000","last_fullsized":"2024-03-26T21:48:23.192320+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.179909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910862+0000","last_change":"2024-03-26T21:47:17.665549+0000","last_active":"2024-03-26T21:48:26.910862+0000","last_peered":"2024-03-26T21:48:26.910862+0000","last_clean":"2024-03-26T21:48:26.910862+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:48:26.910862+0000","last_undegraded":"2024-03-26T21:48:26.910862+0000","last_fullsized":"2024-03-26T21:48:26.910862+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:47:17.665425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":505,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180958+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:48:23.180958+0000","last_peered":"2024-03-26T21:48:23.180958+0000","last_clean":"2024-03-26T21:48:23.180958+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:48:23.180958+0000","last_undegraded":"2024-03-26T21:48:23.180958+0000","last_fullsized":"2024-03-26T21:48:23.180958+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:08.966616+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910362+0000","last_change":"2024-03-26T21:48:03.850832+0000","last_active":"2024-03-26T21:48:26.910362+0000","last_peered":"2024-03-26T21:48:26.910362+0000","last_clean":"2024-03-26T21:48:26.910362+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:48:26.910362+0000","last_undegraded":"2024-03-26T21:48:26.910362+0000","last_fullsized":"2024-03-26T21:48:26.910362+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.850650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.850650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.318466+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":586,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.192041+0000","last_change":"2024-03-26T21:44:49.460726+0000","last_active":"2024-03-26T21:48:23.192041+0000","last_peered":"2024-03-26T21:48:23.192041+0000","last_clean":"2024-03-26T21:48:23.192041+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:48:23.192041+0000","last_undegraded":"2024-03-26T21:48:23.192041+0000","last_fullsized":"2024-03-26T21:48:23.192041+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:44:07.631570+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:44:07.631570+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.911432+0000","last_change":"2024-03-26T21:47:16.612945+0000","last_active":"2024-03-26T21:48:26.911432+0000","last_peered":"2024-03-26T21:48:26.911432+0000","last_clean":"2024-03-26T21:48:26.911432+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:48:26.911432+0000","last_undegraded":"2024-03-26T21:48:26.911432+0000","last_fullsized":"2024-03-26T21:48:26.911432+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:16.612800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:47:16.612800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":839,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-26T21:48:26.910366+0000","last_change":"2024-03-26T21:47:06.579483+0000","last_active":"2024-03-26T21:48:26.910366+0000","last_peered":"2024-03-26T21:48:26.910366+0000","last_clean":"2024-03-26T21:48:26.910366+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:48:26.910366+0000","last_undegraded":"2024-03-26T21:48:26.910366+0000","last_fullsized":"2024-03-26T21:48:26.910366+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:06.579368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:23.539529+0000","last_clean_scrub_stamp":"2024-03-26T21:47:06.579368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":828,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.180168+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:48:23.180168+0000","last_peered":"2024-03-26T21:48:23.180168+0000","last_clean":"2024-03-26T21:48:23.180168+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:48:23.180168+0000","last_undegraded":"2024-03-26T21:48:23.180168+0000","last_fullsized":"2024-03-26T21:48:23.180168+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.767114+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"234'27","reported_seq":196,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919263+0000","last_change":"2024-03-26T21:44:49.489035+0000","last_active":"2024-03-26T21:48:21.919263+0000","last_peered":"2024-03-26T21:48:21.919263+0000","last_clean":"2024-03-26T21:48:21.919263+0000","last_became_active":"2024-03-26T21:44:49.407818+0000","last_became_peered":"2024-03-26T21:44:49.407818+0000","last_unstale":"2024-03-26T21:48:21.919263+0000","last_undegraded":"2024-03-26T21:48:21.919263+0000","last_fullsized":"2024-03-26T21:48:21.919263+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:44:00.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:44:00.432782+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919127+0000","last_change":"2024-03-26T21:47:58.673734+0000","last_active":"2024-03-26T21:48:21.919127+0000","last_peered":"2024-03-26T21:48:21.919127+0000","last_clean":"2024-03-26T21:48:21.919127+0000","last_became_active":"2024-03-26T21:44:49.404120+0000","last_became_peered":"2024-03-26T21:44:49.404120+0000","last_unstale":"2024-03-26T21:48:21.919127+0000","last_undegraded":"2024-03-26T21:48:21.919127+0000","last_fullsized":"2024-03-26T21:48:21.919127+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.724887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":535,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191846+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:48:23.191846+0000","last_peered":"2024-03-26T21:48:23.191846+0000","last_clean":"2024-03-26T21:48:23.191846+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:48:23.191846+0000","last_undegraded":"2024-03-26T21:48:23.191846+0000","last_fullsized":"2024-03-26T21:48:23.191846+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:42.313798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.919900+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:48:21.919900+0000","last_peered":"2024-03-26T21:48:21.919900+0000","last_clean":"2024-03-26T21:48:21.919900+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:48:21.919900+0000","last_undegraded":"2024-03-26T21:48:21.919900+0000","last_fullsized":"2024-03-26T21:48:21.919900+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:52.864285+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":570,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.192013+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:48:23.192013+0000","last_peered":"2024-03-26T21:48:23.192013+0000","last_clean":"2024-03-26T21:48:23.192013+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:48:23.192013+0000","last_undegraded":"2024-03-26T21:48:23.192013+0000","last_fullsized":"2024-03-26T21:48:23.192013+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":832,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.178920+0000","last_change":"2024-03-26T21:46:09.451365+0000","last_active":"2024-03-26T21:48:23.178920+0000","last_peered":"2024-03-26T21:48:23.178920+0000","last_clean":"2024-03-26T21:48:23.178920+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:48:23.178920+0000","last_undegraded":"2024-03-26T21:48:23.178920+0000","last_fullsized":"2024-03-26T21:48:23.178920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:46:09.451237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:46:09.451237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":846,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-03-26T21:48:21.902047+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:48:21.902047+0000","last_peered":"2024-03-26T21:48:21.902047+0000","last_clean":"2024-03-26T21:48:21.902047+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:48:21.902047+0000","last_undegraded":"2024-03-26T21:48:21.902047+0000","last_fullsized":"2024-03-26T21:48:21.902047+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:41.572498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":535,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-03-26T21:48:23.191844+0000","last_change":"2024-03-26T21:47:17.587843+0000","last_active":"2024-03-26T21:48:23.191844+0000","last_peered":"2024-03-26T21:48:23.191844+0000","last_clean":"2024-03-26T21:48:23.191844+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:48:23.191844+0000","last_undegraded":"2024-03-26T21:48:23.191844+0000","last_fullsized":"2024-03-26T21:48:23.191844+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:17.587721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:17.587721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:25.276843+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4089577472,"num_objects":62402,"num_object_clones":0,"num_object_copies":187206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62402,"num_write_kb":3993728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12365529088,"data_stored":12365529088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62402,"ondisk_log_size":62402,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1067,"num_read_kb":1062,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1112,"num_read_kb":869,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973781,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1865036,"kb_used_data":1436676,"kb_used_omap":6,"kb_used_meta":428345,"kb_avail":91880116,"statfs":{"total":95995035648,"available":94085238784,"internally_reserved":0,"allocated":1471156224,"data_stored":1470518161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":438625576},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365049,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2703672,"kb_used_data":2076912,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91041480,"statfs":{"total":95995035648,"available":93226475520,"internally_reserved":0,"allocated":2126757888,"data_stored":2126144825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575122,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2718040,"kb_used_data":1914960,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91027112,"statfs":{"total":95995035648,"available":93211762688,"internally_reserved":0,"allocated":1960919040,"data_stored":1960267486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000864,"kb_used_data":1229716,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":91744288,"statfs":{"total":95995035648,"available":93946150912,"internally_reserved":0,"allocated":1259229184,"data_stored":1258581085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411597,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2250560,"kb_used_data":1573812,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91494592,"statfs":{"total":95995035648,"available":93690462208,"internally_reserved":0,"allocated":1611583488,"data_stored":1610972782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149958,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2596112,"kb_used_data":1698632,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91149040,"statfs":{"total":95995035648,"available":93336616960,"internally_reserved":0,"allocated":1739399168,"data_stored":1738754957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":919009735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650555,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1726680,"kb_used_data":1022608,"kb_used_omap":4,"kb_used_meta":704059,"kb_avail":92018472,"statfs":{"total":95995035648,"available":94226915328,"internally_reserved":0,"allocated":1047150592,"data_stored":1046506169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":720957091},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659388,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729304,"kb_used_data":1052556,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92015848,"statfs":{"total":95995035648,"available":94224228352,"internally_reserved":0,"allocated":1077817344,"data_stored":1077182631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1258749952,"data_stored":1258749952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1970798592,"data_stored":1970798592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1625882624,"data_stored":1625882624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1744633856,"data_stored":1744633856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1055981568,"data_stored":1055981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1080492032,"data_stored":1080492032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2144993280,"data_stored":2144993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1483997184,"data_stored":1483997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:48:30.220 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:48:30.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:30.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:30.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: 6.6 scrub starts 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: 6.6 scrub ok 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:48:30.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:30 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:48:30.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: 6.6 scrub starts 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: 6.6 scrub ok 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:48:30.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: 6.6 scrub starts 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: 6.6 scrub ok 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:48:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:30 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:48:30.577 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:30.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:30.720 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:30.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:30.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 39 16 70314 70298 112.617 73.625 0.00782001 0.00887483 2024-03-26T21:48:30.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:30.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:30.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:31.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:31.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:31.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.284 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:31.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:31.385 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:31.420 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T21:48:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:48:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:48:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:48:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:48:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:48:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:48:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:48:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 5.1b deep-scrub starts 2024-03-26T21:48:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:31 smithi067 ceph-mon[33346]: 5.1b deep-scrub ok 2024-03-26T21:48:31.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:31.560 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:31.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 5.1b deep-scrub starts 2024-03-26T21:48:31.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[29406]: 5.1b deep-scrub ok 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: pgmap v948: 148 pgs: 148 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 5.1b deep-scrub starts 2024-03-26T21:48:31.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:31 smithi055 ceph-mon[40587]: 5.1b deep-scrub ok 2024-03-26T21:48:31.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:48:31.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:31.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:31.790 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:31.792 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:31.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:31.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:31.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:48:31.913161+0000 min lat: 0.00133566 max lat: 0.179135 avg lat: 0.00894318 2024-03-26T21:48:31.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:48:31.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 40 16 71551 71535 111.734 77.3125 0.0195697 0.00894318 2024-03-26T21:48:31.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.026 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:32.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:32.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:32.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:32.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:32.296 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:32.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:32.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:32 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:48:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:32 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:48:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:32 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:48:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:32 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:48:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:32 smithi067 ceph-mon[33346]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:48:32.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:48:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:48:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:48:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:48:32.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[29406]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:48:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:48:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:48:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:48:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:48:32.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:32 smithi055 ceph-mon[40587]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T21:48:32.573 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:32.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:32.712 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:32.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:32.856 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:32.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 41 16 72796 72780 110.907 77.8125 0.0101828 0.00901074 2024-03-26T21:48:32.958 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:32.996 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:32.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:33.043 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:48:33.044 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:48:33.096 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:33.144 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:33.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:33.292 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:33.394 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:33.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: pgmap v950: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: 5.5 deep-scrub starts 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: 5.5 deep-scrub ok 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:33 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:33.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: pgmap v950: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: 5.5 deep-scrub starts 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: 5.5 deep-scrub ok 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: pgmap v950: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s 2024-03-26T21:48:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: 5.5 deep-scrub starts 2024-03-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: 5.5 deep-scrub ok 2024-03-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:33 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:33.572 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:33.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:33.811 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.851 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:33.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 42 16 74050 74034 110.132 78.375 0.0131999 0.00907541 2024-03-26T21:48:33.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:33.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:33.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:34.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:34.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:34.234 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:34.271 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:48:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.287 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:34.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:34.333 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:48:34.333 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:48:34.333 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:48:34.334 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:48:34.334 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:48:34.334 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:48:34.334 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:48:34.334 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:34 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:48:34.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:34.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:34 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:48:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:34 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:48:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:34 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:48:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:34 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:48:34.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:34.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:34.702 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:34.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:34.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:34.851 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:34.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 43 16 75259 75243 109.328 75.5625 0.00997308 0.00914117 2024-03-26T21:48:34.928 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:34.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:34.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:35.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:35.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:35.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: pgmap v952: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:48:35.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:48:35.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:48:35.300 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: pgmap v952: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:35.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:48:35.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:48:35.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:48:35.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:48:35.302 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:35 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:48:35.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:35.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: pgmap v952: 148 pgs: 148 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T21:48:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:48:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:48:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T21:48:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:48:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:48:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:48:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:35 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:48:35.519 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:35.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:35.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:35.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:35.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:35.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 44 16 76503 76487 108.611 77.75 0.018913 0.00920195 2024-03-26T21:48:35.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:35.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:35.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:36.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:36.084 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:36.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:36.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[40587]: 3.e deep-scrub starts 2024-03-26T21:48:36.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[40587]: 3.e deep-scrub ok 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[29406]: 3.e deep-scrub starts 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[29406]: 3.e deep-scrub ok 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:48:36.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:36 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:48:36.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.354 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:36.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:36.455 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:36.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:36.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:36.621 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:36.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:36 smithi067 ceph-mon[33346]: 3.e deep-scrub starts 2024-03-26T21:48:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:36 smithi067 ceph-mon[33346]: 3.e deep-scrub ok 2024-03-26T21:48:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:36 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:48:36.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:36 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:48:36.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:36.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:36.886 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:36.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:36.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 45 16 77709 77693 107.872 75.375 0.0193271 0.00926393 2024-03-26T21:48:36.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:37.020 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:37.122 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:37.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:37.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:37.296 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:37.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:37.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: pgmap v954: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T21:48:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:48:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: 4.9 deep-scrub starts 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: 4.9 deep-scrub ok 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[40587]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: pgmap v954: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:48:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:37.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:37.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: 4.9 deep-scrub starts 2024-03-26T21:48:37.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: 4.9 deep-scrub ok 2024-03-26T21:48:37.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:48:37.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:37 smithi055 ceph-mon[29406]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:48:37.571 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: pgmap v954: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: 4.9 deep-scrub starts 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: 4.9 deep-scrub ok 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:48:37.614 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-mon[33346]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T21:48:37.673 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:37.710 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:37.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:37.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 46 15 78928 78913 107.185 76.25 0.0134379 0.00932502 2024-03-26T21:48:37.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:37.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:37.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:38.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:38.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:38.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:38.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:38.329 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:38.367 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:38.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:38.511 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:38.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:48:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[29406]: 3.10 deep-scrub starts 2024-03-26T21:48:38.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[29406]: 3.10 deep-scrub ok 2024-03-26T21:48:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:48:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[40587]: 3.10 deep-scrub starts 2024-03-26T21:48:38.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-mon[40587]: 3.10 deep-scrub ok 2024-03-26T21:48:38.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:38.649 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:38 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:48:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:38 smithi067 ceph-mon[33346]: 3.10 deep-scrub starts 2024-03-26T21:48:38.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:38 smithi067 ceph-mon[33346]: 3.10 deep-scrub ok 2024-03-26T21:48:38.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:38.789 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:38.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:38.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 47 16 80024 80008 106.36 68.4375 0.0165701 0.00939731 2024-03-26T21:48:38.927 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:38.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:39.029 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:39.067 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:39.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:39.255 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:39.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:39.335 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:39.437 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:39.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:39.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[29406]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[40587]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:48:39.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:39 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:48:39.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:39.619 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:39 smithi067 ceph-mon[33346]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T21:48:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:39 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:48:39.640 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:39 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:48:39.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.760 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:39.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:39.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:39.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:39.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:39.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 48 16 81203 81187 105.68 73.6875 0.00829289 0.00945733 2024-03-26T21:48:39.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:40.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:40.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:40.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:40.321 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:40.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:40.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:40.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:40.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:40 smithi055 ceph-mon[29406]: 3.14 deep-scrub starts 2024-03-26T21:48:40.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:40 smithi055 ceph-mon[29406]: 3.14 deep-scrub ok 2024-03-26T21:48:40.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:40 smithi055 ceph-mon[40587]: 3.14 deep-scrub starts 2024-03-26T21:48:40.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:40 smithi055 ceph-mon[40587]: 3.14 deep-scrub ok 2024-03-26T21:48:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:40.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:40.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:40 smithi067 ceph-mon[33346]: 3.14 deep-scrub starts 2024-03-26T21:48:40.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:40 smithi067 ceph-mon[33346]: 3.14 deep-scrub ok 2024-03-26T21:48:40.819 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:40.857 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:40.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:40.905 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 49 16 82348 82332 104.983 71.5625 0.00642187 0.00952078 2024-03-26T21:48:40.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:41.001 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:41.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:41.143 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:41.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:41.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:41.417 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:41.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:41.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[40587]: pgmap v957: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T21:48:41.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[40587]: 6.1 deep-scrub starts 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[40587]: 6.1 deep-scrub ok 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[29406]: pgmap v957: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[29406]: 6.1 deep-scrub starts 2024-03-26T21:48:41.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:41 smithi055 ceph-mon[29406]: 6.1 deep-scrub ok 2024-03-26T21:48:41.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.695 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:41.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:41 smithi067 ceph-mon[33346]: pgmap v957: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-26T21:48:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:41 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:48:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:41 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:48:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:41 smithi067 ceph-mon[33346]: 6.1 deep-scrub starts 2024-03-26T21:48:41.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:41 smithi067 ceph-mon[33346]: 6.1 deep-scrub ok 2024-03-26T21:48:41.767 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:41.796 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:41.835 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:41.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 50 16 83492 83476 104.313 71.5 0.00748571 0.00958125 2024-03-26T21:48:41.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:41.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:42.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.114 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:42.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:42.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.252 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:42.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:42.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:42.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:42.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:42.552 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:42.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:48:42.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 3.9 deep-scrub starts 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 3.9 deep-scrub ok 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:48:42.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[40587]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:48:42.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 3.9 deep-scrub starts 2024-03-26T21:48:42.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 3.9 deep-scrub ok 2024-03-26T21:48:42.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:48:42.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:48:42.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:42 smithi055 ceph-mon[29406]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:48:42.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 3.9 deep-scrub starts 2024-03-26T21:48:42.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 3.9 deep-scrub ok 2024-03-26T21:48:42.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:48:42.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:48:42.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:42 smithi067 ceph-mon[33346]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T21:48:42.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:42.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:42.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 51 15 84667 84652 103.709 73.5 0.0170598 0.00963709 2024-03-26T21:48:42.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:42.950 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:42.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:43.052 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:43.092 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:43.194 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:43.232 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:43.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:43.372 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:43.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:43.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[29406]: pgmap v959: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 683 B/s rd, 54 MiB/s wr, 865 op/s 2024-03-26T21:48:43.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:48:43.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:48:43.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[40587]: pgmap v959: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 683 B/s rd, 54 MiB/s wr, 865 op/s 2024-03-26T21:48:43.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:48:43.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:43 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:48:43.616 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:43.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:43 smithi067 ceph-mon[33346]: pgmap v959: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 683 B/s rd, 54 MiB/s wr, 865 op/s 2024-03-26T21:48:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:43 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:48:43.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:43 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:48:43.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.804 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:43.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:43.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:43.907 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 52 16 85828 85812 103.109 72.5 0.0153605 0.00969422 2024-03-26T21:48:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:43.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:43.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:44.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:44.087 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:44.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:44.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:44.293 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:48:44.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:44.351 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:48:44.352 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:44.408 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:48:44.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:44.480 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:48:44.487 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:44.529 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:48:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[29406]: 2.2 deep-scrub starts 2024-03-26T21:48:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[29406]: 2.2 deep-scrub ok 2024-03-26T21:48:44.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2443881005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:44.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[40587]: 2.2 deep-scrub starts 2024-03-26T21:48:44.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[40587]: 2.2 deep-scrub ok 2024-03-26T21:48:44.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2443881005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:44.577 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:48:44.589 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:44.629 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:48:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:44.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:44.682 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:48:44.686 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:48:44.686 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:48:44.686 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:48:44.687 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:48:44.687 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:48:44.687 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:48:44.687 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:48:44.687 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:48:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:44 smithi067 ceph-mon[33346]: 2.2 deep-scrub starts 2024-03-26T21:48:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:44 smithi067 ceph-mon[33346]: 2.2 deep-scrub ok 2024-03-26T21:48:44.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2443881005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:44.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.791 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:44.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:44.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:44.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 53 16 86648 86632 102.131 51.25 0.0651181 0.00978372 2024-03-26T21:48:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:44.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:45.025 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:45.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:45.162 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:48:45.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:45.169 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:48:45.195 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:45.297 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:45.338 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:45.359 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:48:45.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:45.461 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: pgmap v961: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 5.0 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-26T21:48:45.461 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 11.d scrub starts 2024-03-26T21:48:45.461 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2395833160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 3.d deep-scrub starts 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 3.d deep-scrub ok 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 6.1a scrub starts 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[29406]: 6.1a scrub ok 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: pgmap v961: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 5.0 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 11.d scrub starts 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:48:45.462 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2395833160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 3.d deep-scrub starts 2024-03-26T21:48:45.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 3.d deep-scrub ok 2024-03-26T21:48:45.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 6.1a scrub starts 2024-03-26T21:48:45.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:45 smithi055 ceph-mon[40587]: 6.1a scrub ok 2024-03-26T21:48:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:45.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.620 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:45.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:45.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: pgmap v961: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 5.0 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 11.d scrub starts 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2395833160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:48:45.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 3.d deep-scrub starts 2024-03-26T21:48:45.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 3.d deep-scrub ok 2024-03-26T21:48:45.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 6.1a scrub starts 2024-03-26T21:48:45.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:45 smithi067 ceph-mon[33346]: 6.1a scrub ok 2024-03-26T21:48:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:45.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:45.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:45.895 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:45.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:45.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 54 16 87714 87698 101.473 66.625 0.0265099 0.00984856 2024-03-26T21:48:45.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.035 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:46.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.162 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:46.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:46.296 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:46.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:46.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:46.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:46.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: 11.2 scrub starts 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: 5.1e deep-scrub starts 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: 5.1e deep-scrub ok 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: 11.3 scrub starts 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:48:46.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:48:46.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:46 smithi067 ceph-mon[33346]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:48:46.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: 11.2 scrub starts 2024-03-26T21:48:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: 5.1e deep-scrub starts 2024-03-26T21:48:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: 5.1e deep-scrub ok 2024-03-26T21:48:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: 11.3 scrub starts 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[29406]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: 11.2 scrub starts 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: 5.1e deep-scrub starts 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: 5.1e deep-scrub ok 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: 11.3 scrub starts 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:48:46.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:46 smithi055 ceph-mon[40587]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T21:48:46.819 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:46.907 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 55 16 88717 88701 100.768 62.6875 0.0198503 0.00991849 2024-03-26T21:48:46.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:46.959 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:46.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:47.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:47.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:47.207 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.244 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:47.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:47.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:47.484 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.527 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:47.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:47.624 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:47 smithi067 ceph-mon[33346]: pgmap v963: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 60 MiB/s wr, 980 op/s 2024-03-26T21:48:47.624 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:47 smithi067 ceph-mon[33346]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:48:47.629 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:47.673 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:47.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:47.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:47 smithi055 ceph-mon[29406]: pgmap v963: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 60 MiB/s wr, 980 op/s 2024-03-26T21:48:47.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:47 smithi055 ceph-mon[29406]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:48:47.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:47 smithi055 ceph-mon[40587]: pgmap v963: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 60 MiB/s wr, 980 op/s 2024-03-26T21:48:47.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:47 smithi055 ceph-mon[40587]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T21:48:47.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 56 16 89817 89801 100.196 68.75 0.00589221 0.00997575 2024-03-26T21:48:47.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:47.944 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:47.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:47.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:48.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:48.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:48.176 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.221 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:48.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:48.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:48.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:48.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:48.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:48.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.639 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:48.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:48:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:48:48.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:48.767 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:48.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:48.903 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:48.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:48.906 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 57 16 90845 90829 99.5656 64.25 0.0150675 0.0100382 2024-03-26T21:48:49.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:49.040 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":961,"stamp":"2024-03-26T21:48:43.731678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5514009000,"num_objects":84326,"num_object_clones":0,"num_object_copies":252978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84326,"num_whiteouts":0,"num_read":2428,"num_read_kb":2276,"num_write":85045,"num_write_kb":5386653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":1777738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81663,"ondisk_log_size":81663,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21701368,"kb_used_data":16019796,"kb_used_omap":20,"kb_used_meta":5681451,"kb_avail":728259848,"statfs":{"total":767960285184,"available":745738084352,"internally_reserved":0,"allocated":16404271104,"data_stored":16398912216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20884,"internal_metadata":5817806444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":730791936,"num_objects":11151,"num_object_clones":0,"num_object_copies":33453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11151,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":11167,"num_write_kb":713664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.825646"},"pg_stats":[{"pgid":"11.3","version":"455'5238","reported_seq":5289,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.902468+0000","last_change":"2024-03-26T21:47:39.050006+0000","last_active":"2024-03-26T21:48:41.902468+0000","last_peered":"2024-03-26T21:48:41.902468+0000","last_clean":"2024-03-26T21:48:41.902468+0000","last_became_active":"2024-03-26T21:47:39.049657+0000","last_became_peered":"2024-03-26T21:47:39.049657+0000","last_unstale":"2024-03-26T21:48:41.902468+0000","last_undegraded":"2024-03-26T21:48:41.902468+0000","last_fullsized":"2024-03-26T21:48:41.902468+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.260841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343277568,"num_objects":5238,"num_object_clones":0,"num_object_copies":15714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":335232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"234'25","reported_seq":627,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907612+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:48:36.907612+0000","last_peered":"2024-03-26T21:48:36.907612+0000","last_clean":"2024-03-26T21:48:36.907612+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:48:36.907612+0000","last_undegraded":"2024-03-26T21:48:36.907612+0000","last_fullsized":"2024-03-26T21:48:36.907612+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.420796+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927203+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:48:41.927203+0000","last_peered":"2024-03-26T21:48:41.927203+0000","last_clean":"2024-03-26T21:48:41.927203+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:48:41.927203+0000","last_undegraded":"2024-03-26T21:48:41.927203+0000","last_fullsized":"2024-03-26T21:48:41.927203+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"277'11","reported_seq":1123,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916328+0000","last_change":"2024-03-26T21:48:03.066076+0000","last_active":"2024-03-26T21:48:41.916328+0000","last_peered":"2024-03-26T21:48:41.916328+0000","last_clean":"2024-03-26T21:48:41.916328+0000","last_became_active":"2024-03-26T21:41:25.611530+0000","last_became_peered":"2024-03-26T21:41:25.611530+0000","last_unstale":"2024-03-26T21:48:41.916328+0000","last_undegraded":"2024-03-26T21:48:41.916328+0000","last_fullsized":"2024-03-26T21:48:41.916328+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.029799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":546,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907678+0000","last_change":"2024-03-26T21:47:25.630901+0000","last_active":"2024-03-26T21:48:36.907678+0000","last_peered":"2024-03-26T21:48:36.907678+0000","last_clean":"2024-03-26T21:48:36.907678+0000","last_became_active":"2024-03-26T21:36:51.692843+0000","last_became_peered":"2024-03-26T21:36:51.692843+0000","last_unstale":"2024-03-26T21:48:36.907678+0000","last_undegraded":"2024-03-26T21:48:36.907678+0000","last_fullsized":"2024-03-26T21:48:36.907678+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:48.426306+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"456'5352","reported_seq":5408,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.623948+0000","last_change":"2024-03-26T21:47:39.048799+0000","last_active":"2024-03-26T21:48:42.623948+0000","last_peered":"2024-03-26T21:48:42.623948+0000","last_clean":"2024-03-26T21:48:42.623948+0000","last_became_active":"2024-03-26T21:47:39.048454+0000","last_became_peered":"2024-03-26T21:47:39.048454+0000","last_unstale":"2024-03-26T21:48:42.623948+0000","last_undegraded":"2024-03-26T21:48:42.623948+0000","last_fullsized":"2024-03-26T21:48:42.623948+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":342528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"282'39","reported_seq":593,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907616+0000","last_change":"2024-03-26T21:47:23.550040+0000","last_active":"2024-03-26T21:48:36.907616+0000","last_peered":"2024-03-26T21:48:36.907616+0000","last_clean":"2024-03-26T21:48:36.907616+0000","last_became_active":"2024-03-26T21:36:51.704908+0000","last_became_peered":"2024-03-26T21:36:51.704908+0000","last_unstale":"2024-03-26T21:48:36.907616+0000","last_undegraded":"2024-03-26T21:48:36.907616+0000","last_fullsized":"2024-03-26T21:48:36.907616+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:51.983719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":889,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.303753+0000","last_change":"2024-03-26T21:47:24.976202+0000","last_active":"2024-03-26T21:48:41.303753+0000","last_peered":"2024-03-26T21:48:41.303753+0000","last_clean":"2024-03-26T21:48:41.303753+0000","last_became_active":"2024-03-26T21:36:51.711618+0000","last_became_peered":"2024-03-26T21:36:51.711618+0000","last_unstale":"2024-03-26T21:48:41.303753+0000","last_undegraded":"2024-03-26T21:48:41.303753+0000","last_fullsized":"2024-03-26T21:48:41.303753+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:53.571891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":883,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916550+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:48:41.916550+0000","last_peered":"2024-03-26T21:48:41.916550+0000","last_clean":"2024-03-26T21:48:41.916550+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:48:41.916550+0000","last_undegraded":"2024-03-26T21:48:41.916550+0000","last_fullsized":"2024-03-26T21:48:41.916550+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:04.817836+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":890,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916652+0000","last_change":"2024-03-26T21:48:05.290034+0000","last_active":"2024-03-26T21:48:41.916652+0000","last_peered":"2024-03-26T21:48:41.916652+0000","last_clean":"2024-03-26T21:48:41.916652+0000","last_became_active":"2024-03-26T21:44:49.410905+0000","last_became_peered":"2024-03-26T21:44:49.410905+0000","last_unstale":"2024-03-26T21:48:41.916652+0000","last_undegraded":"2024-03-26T21:48:41.916652+0000","last_fullsized":"2024-03-26T21:48:41.916652+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.592759+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"455'5109","reported_seq":5163,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.884009+0000","last_change":"2024-03-26T21:47:39.047939+0000","last_active":"2024-03-26T21:48:41.884009+0000","last_peered":"2024-03-26T21:48:41.884009+0000","last_clean":"2024-03-26T21:48:41.884009+0000","last_became_active":"2024-03-26T21:47:39.047535+0000","last_became_peered":"2024-03-26T21:47:39.047535+0000","last_unstale":"2024-03-26T21:48:41.884009+0000","last_undegraded":"2024-03-26T21:48:41.884009+0000","last_fullsized":"2024-03-26T21:48:41.884009+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:47.698282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334823424,"num_objects":5109,"num_object_clones":0,"num_object_copies":15327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5109,"num_write_kb":326976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"426'24","reported_seq":575,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907783+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:48:36.907783+0000","last_peered":"2024-03-26T21:48:36.907783+0000","last_clean":"2024-03-26T21:48:36.907783+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:48:36.907783+0000","last_undegraded":"2024-03-26T21:48:36.907783+0000","last_fullsized":"2024-03-26T21:48:36.907783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:52.521281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":547,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907080+0000","last_change":"2024-03-26T21:47:15.544701+0000","last_active":"2024-03-26T21:48:36.907080+0000","last_peered":"2024-03-26T21:48:36.907080+0000","last_clean":"2024-03-26T21:48:36.907080+0000","last_became_active":"2024-03-26T21:41:25.613187+0000","last_became_peered":"2024-03-26T21:41:25.613187+0000","last_unstale":"2024-03-26T21:48:36.907080+0000","last_undegraded":"2024-03-26T21:48:36.907080+0000","last_fullsized":"2024-03-26T21:48:36.907080+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:15.544567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.544567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":864,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.911273+0000","last_change":"2024-03-26T21:48:16.730412+0000","last_active":"2024-03-26T21:48:41.911273+0000","last_peered":"2024-03-26T21:48:41.911273+0000","last_clean":"2024-03-26T21:48:41.911273+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:48:41.911273+0000","last_undegraded":"2024-03-26T21:48:41.911273+0000","last_fullsized":"2024-03-26T21:48:41.911273+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.730303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.730303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:35.540578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927978+0000","last_change":"2024-03-26T21:48:30.759102+0000","last_active":"2024-03-26T21:48:41.927978+0000","last_peered":"2024-03-26T21:48:41.927978+0000","last_clean":"2024-03-26T21:48:41.927978+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:48:41.927978+0000","last_undegraded":"2024-03-26T21:48:41.927978+0000","last_fullsized":"2024-03-26T21:48:41.927978+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.758986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.758986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:36.747058+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"456'5282","reported_seq":5338,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.043742+0000","last_change":"2024-03-26T21:47:39.053277+0000","last_active":"2024-03-26T21:48:43.043742+0000","last_peered":"2024-03-26T21:48:43.043742+0000","last_clean":"2024-03-26T21:48:43.043742+0000","last_became_active":"2024-03-26T21:47:39.052942+0000","last_became_peered":"2024-03-26T21:47:39.052942+0000","last_unstale":"2024-03-26T21:48:43.043742+0000","last_undegraded":"2024-03-26T21:48:43.043742+0000","last_fullsized":"2024-03-26T21:48:43.043742+0000","mapping_epoch":433,"log_start":"440'300","ondisk_log_start":"440'300","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.415225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346161152,"num_objects":5282,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5282,"num_write_kb":338048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'29","reported_seq":1004,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.937924+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:48:42.937924+0000","last_peered":"2024-03-26T21:48:42.937924+0000","last_clean":"2024-03-26T21:48:42.937924+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:48:42.937924+0000","last_undegraded":"2024-03-26T21:48:42.937924+0000","last_fullsized":"2024-03-26T21:48:42.937924+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.150388+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":38,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927491+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:48:41.927491+0000","last_peered":"2024-03-26T21:48:41.927491+0000","last_clean":"2024-03-26T21:48:41.927491+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:48:41.927491+0000","last_undegraded":"2024-03-26T21:48:41.927491+0000","last_fullsized":"2024-03-26T21:48:41.927491+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.420341+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":701,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907930+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:48:36.907930+0000","last_peered":"2024-03-26T21:48:36.907930+0000","last_clean":"2024-03-26T21:48:36.907930+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:48:36.907930+0000","last_undegraded":"2024-03-26T21:48:36.907930+0000","last_fullsized":"2024-03-26T21:48:36.907930+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.930604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":847,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910273+0000","last_change":"2024-03-26T21:48:19.935968+0000","last_active":"2024-03-26T21:48:41.910273+0000","last_peered":"2024-03-26T21:48:41.910273+0000","last_clean":"2024-03-26T21:48:41.910273+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:48:41.910273+0000","last_undegraded":"2024-03-26T21:48:41.910273+0000","last_fullsized":"2024-03-26T21:48:41.910273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.935613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.935613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.386876+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"456'5259","reported_seq":5315,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.316994+0000","last_change":"2024-03-26T21:47:39.051954+0000","last_active":"2024-03-26T21:48:43.316994+0000","last_peered":"2024-03-26T21:48:43.316994+0000","last_clean":"2024-03-26T21:48:43.316994+0000","last_became_active":"2024-03-26T21:47:39.051498+0000","last_became_peered":"2024-03-26T21:47:39.051498+0000","last_unstale":"2024-03-26T21:48:43.316994+0000","last_undegraded":"2024-03-26T21:48:43.316994+0000","last_fullsized":"2024-03-26T21:48:43.316994+0000","mapping_epoch":433,"log_start":"440'100","ondisk_log_start":"440'100","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:01.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344653824,"num_objects":5259,"num_object_clones":0,"num_object_copies":15777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":336576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":565,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907879+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:48:36.907879+0000","last_peered":"2024-03-26T21:48:36.907879+0000","last_clean":"2024-03-26T21:48:36.907879+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:48:36.907879+0000","last_undegraded":"2024-03-26T21:48:36.907879+0000","last_fullsized":"2024-03-26T21:48:36.907879+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.810891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'40","reported_seq":225,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.371476+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:48:42.371476+0000","last_peered":"2024-03-26T21:48:42.371476+0000","last_clean":"2024-03-26T21:48:42.371476+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:48:42.371476+0000","last_undegraded":"2024-03-26T21:48:42.371476+0000","last_fullsized":"2024-03-26T21:48:42.371476+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:30.618938+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":42,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927226+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:48:41.927226+0000","last_peered":"2024-03-26T21:48:41.927226+0000","last_clean":"2024-03-26T21:48:41.927226+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:48:41.927226+0000","last_undegraded":"2024-03-26T21:48:41.927226+0000","last_fullsized":"2024-03-26T21:48:41.927226+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:18.541710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":566,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907532+0000","last_change":"2024-03-26T21:48:28.651785+0000","last_active":"2024-03-26T21:48:36.907532+0000","last_peered":"2024-03-26T21:48:36.907532+0000","last_clean":"2024-03-26T21:48:36.907532+0000","last_became_active":"2024-03-26T21:44:49.394613+0000","last_became_peered":"2024-03-26T21:44:49.394613+0000","last_unstale":"2024-03-26T21:48:36.907532+0000","last_undegraded":"2024-03-26T21:48:36.907532+0000","last_fullsized":"2024-03-26T21:48:36.907532+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:28.651656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:48:28.651656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:35.210781+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"454'5135","reported_seq":5187,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.171899+0000","last_change":"2024-03-26T21:47:39.053658+0000","last_active":"2024-03-26T21:48:41.171899+0000","last_peered":"2024-03-26T21:48:41.171899+0000","last_clean":"2024-03-26T21:48:41.171899+0000","last_became_active":"2024-03-26T21:47:39.053244+0000","last_became_peered":"2024-03-26T21:47:39.053244+0000","last_unstale":"2024-03-26T21:48:41.171899+0000","last_undegraded":"2024-03-26T21:48:41.171899+0000","last_fullsized":"2024-03-26T21:48:41.171899+0000","mapping_epoch":433,"log_start":"440'200","ondisk_log_start":"440'200","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:58.455414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336527360,"num_objects":5135,"num_object_clones":0,"num_object_copies":15405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5135,"num_write_kb":328640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":255,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927601+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:48:41.927601+0000","last_peered":"2024-03-26T21:48:41.927601+0000","last_clean":"2024-03-26T21:48:41.927601+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:48:41.927601+0000","last_undegraded":"2024-03-26T21:48:41.927601+0000","last_fullsized":"2024-03-26T21:48:41.927601+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"234'28","reported_seq":583,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907678+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:48:36.907678+0000","last_peered":"2024-03-26T21:48:36.907678+0000","last_clean":"2024-03-26T21:48:36.907678+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:48:36.907678+0000","last_undegraded":"2024-03-26T21:48:36.907678+0000","last_fullsized":"2024-03-26T21:48:36.907678+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:46.571871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":864,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902862+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:48:36.902862+0000","last_peered":"2024-03-26T21:48:36.902862+0000","last_clean":"2024-03-26T21:48:36.902862+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:48:36.902862+0000","last_undegraded":"2024-03-26T21:48:36.902862+0000","last_fullsized":"2024-03-26T21:48:36.902862+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.168894+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":773,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.917812+0000","last_change":"2024-03-26T21:48:11.992462+0000","last_active":"2024-03-26T21:48:41.917812+0000","last_peered":"2024-03-26T21:48:41.917812+0000","last_clean":"2024-03-26T21:48:41.917812+0000","last_became_active":"2024-03-26T21:41:25.621973+0000","last_became_peered":"2024-03-26T21:41:25.621973+0000","last_unstale":"2024-03-26T21:48:41.917812+0000","last_undegraded":"2024-03-26T21:48:41.917812+0000","last_fullsized":"2024-03-26T21:48:41.917812+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:31.860005+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"456'5200","reported_seq":5256,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.627513+0000","last_change":"2024-03-26T21:47:39.046470+0000","last_active":"2024-03-26T21:48:42.627513+0000","last_peered":"2024-03-26T21:48:42.627513+0000","last_clean":"2024-03-26T21:48:42.627513+0000","last_became_active":"2024-03-26T21:47:39.046152+0000","last_became_peered":"2024-03-26T21:47:39.046152+0000","last_unstale":"2024-03-26T21:48:42.627513+0000","last_undegraded":"2024-03-26T21:48:42.627513+0000","last_fullsized":"2024-03-26T21:48:42.627513+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5200,"log_dups_size":0,"ondisk_log_size":5200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340787200,"num_objects":5200,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5200,"num_write_kb":332800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1091,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.904234+0000","last_change":"2024-03-26T21:48:32.957198+0000","last_active":"2024-03-26T21:48:36.904234+0000","last_peered":"2024-03-26T21:48:36.904234+0000","last_clean":"2024-03-26T21:48:36.904234+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:48:36.904234+0000","last_undegraded":"2024-03-26T21:48:36.904234+0000","last_fullsized":"2024-03-26T21:48:36.904234+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:48:32.957059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.957059+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:33.666892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'24","reported_seq":939,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.019185+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:48:43.019185+0000","last_peered":"2024-03-26T21:48:43.019185+0000","last_clean":"2024-03-26T21:48:43.019185+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:48:43.019185+0000","last_undegraded":"2024-03-26T21:48:43.019185+0000","last_fullsized":"2024-03-26T21:48:43.019185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:20.573732+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":872,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910608+0000","last_change":"2024-03-26T21:48:00.661315+0000","last_active":"2024-03-26T21:48:41.910608+0000","last_peered":"2024-03-26T21:48:41.910608+0000","last_clean":"2024-03-26T21:48:41.910608+0000","last_became_active":"2024-03-26T21:44:49.403379+0000","last_became_peered":"2024-03-26T21:44:49.403379+0000","last_unstale":"2024-03-26T21:48:41.910608+0000","last_undegraded":"2024-03-26T21:48:41.910608+0000","last_fullsized":"2024-03-26T21:48:41.910608+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:00.661167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:48:00.661167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:03.433855+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":850,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.901237+0000","last_change":"2024-03-26T21:48:30.031376+0000","last_active":"2024-03-26T21:48:36.901237+0000","last_peered":"2024-03-26T21:48:36.901237+0000","last_clean":"2024-03-26T21:48:36.901237+0000","last_became_active":"2024-03-26T21:44:49.436816+0000","last_became_peered":"2024-03-26T21:44:49.436816+0000","last_unstale":"2024-03-26T21:48:36.901237+0000","last_undegraded":"2024-03-26T21:48:36.901237+0000","last_fullsized":"2024-03-26T21:48:36.901237+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:54.267947+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"456'5352","reported_seq":5408,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.001137+0000","last_change":"2024-03-26T21:47:39.051081+0000","last_active":"2024-03-26T21:48:43.001137+0000","last_peered":"2024-03-26T21:48:43.001137+0000","last_clean":"2024-03-26T21:48:43.001137+0000","last_became_active":"2024-03-26T21:47:39.050784+0000","last_became_peered":"2024-03-26T21:47:39.050784+0000","last_unstale":"2024-03-26T21:48:43.001137+0000","last_undegraded":"2024-03-26T21:48:43.001137+0000","last_fullsized":"2024-03-26T21:48:43.001137+0000","mapping_epoch":433,"log_start":"440'400","ondisk_log_start":"440'400","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:44.939822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":342528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":860,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.304229+0000","last_change":"2024-03-26T21:48:02.087751+0000","last_active":"2024-03-26T21:48:41.304229+0000","last_peered":"2024-03-26T21:48:41.304229+0000","last_clean":"2024-03-26T21:48:41.304229+0000","last_became_active":"2024-03-26T21:28:38.868039+0000","last_became_peered":"2024-03-26T21:28:38.868039+0000","last_unstale":"2024-03-26T21:48:41.304229+0000","last_undegraded":"2024-03-26T21:48:41.304229+0000","last_fullsized":"2024-03-26T21:48:41.304229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.376172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"234'29","reported_seq":906,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.905976+0000","last_change":"2024-03-26T21:48:35.997876+0000","last_active":"2024-03-26T21:48:36.905976+0000","last_peered":"2024-03-26T21:48:36.905976+0000","last_clean":"2024-03-26T21:48:36.905976+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:48:36.905976+0000","last_undegraded":"2024-03-26T21:48:36.905976+0000","last_fullsized":"2024-03-26T21:48:36.905976+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:48:35.997722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.997722+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:48.429570+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":878,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916399+0000","last_change":"2024-03-26T21:48:19.765008+0000","last_active":"2024-03-26T21:48:41.916399+0000","last_peered":"2024-03-26T21:48:41.916399+0000","last_clean":"2024-03-26T21:48:41.916399+0000","last_became_active":"2024-03-26T21:28:34.849097+0000","last_became_peered":"2024-03-26T21:28:34.849097+0000","last_unstale":"2024-03-26T21:48:41.916399+0000","last_undegraded":"2024-03-26T21:48:41.916399+0000","last_fullsized":"2024-03-26T21:48:41.916399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.881328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":858,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915908+0000","last_change":"2024-03-26T21:48:40.809436+0000","last_active":"2024-03-26T21:48:41.915908+0000","last_peered":"2024-03-26T21:48:41.915908+0000","last_clean":"2024-03-26T21:48:41.915908+0000","last_became_active":"2024-03-26T21:28:41.116021+0000","last_became_peered":"2024-03-26T21:28:41.116021+0000","last_unstale":"2024-03-26T21:48:41.915908+0000","last_undegraded":"2024-03-26T21:48:41.915908+0000","last_fullsized":"2024-03-26T21:48:41.915908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.809302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:56.878782+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"456'5322","reported_seq":5378,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.991594+0000","last_change":"2024-03-26T21:47:39.047854+0000","last_active":"2024-03-26T21:48:42.991594+0000","last_peered":"2024-03-26T21:48:42.991594+0000","last_clean":"2024-03-26T21:48:42.991594+0000","last_became_active":"2024-03-26T21:47:39.047314+0000","last_became_peered":"2024-03-26T21:47:39.047314+0000","last_unstale":"2024-03-26T21:48:42.991594+0000","last_undegraded":"2024-03-26T21:48:42.991594+0000","last_fullsized":"2024-03-26T21:48:42.991594+0000","mapping_epoch":433,"log_start":"440'500","ondisk_log_start":"440'500","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:07.969650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348782592,"num_objects":5322,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5322,"num_write_kb":340608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":852,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.911352+0000","last_change":"2024-03-26T21:47:59.681466+0000","last_active":"2024-03-26T21:48:41.911352+0000","last_peered":"2024-03-26T21:48:41.911352+0000","last_clean":"2024-03-26T21:48:41.911352+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:48:41.911352+0000","last_undegraded":"2024-03-26T21:48:41.911352+0000","last_fullsized":"2024-03-26T21:48:41.911352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:59.681319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.681319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.002135+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"234'24","reported_seq":902,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902777+0000","last_change":"2024-03-26T21:48:11.911961+0000","last_active":"2024-03-26T21:48:36.902777+0000","last_peered":"2024-03-26T21:48:36.902777+0000","last_clean":"2024-03-26T21:48:36.902777+0000","last_became_active":"2024-03-26T21:28:59.678570+0000","last_became_peered":"2024-03-26T21:28:59.678570+0000","last_unstale":"2024-03-26T21:48:36.902777+0000","last_undegraded":"2024-03-26T21:48:36.902777+0000","last_fullsized":"2024-03-26T21:48:36.902777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:36.106001+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":1915,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914010+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:48:41.914010+0000","last_peered":"2024-03-26T21:48:41.914010+0000","last_clean":"2024-03-26T21:48:41.914010+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:48:41.914010+0000","last_undegraded":"2024-03-26T21:48:41.914010+0000","last_fullsized":"2024-03-26T21:48:41.914010+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:38.910539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1087,"num_read_kb":1082,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":875,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.903483+0000","last_change":"2024-03-26T21:47:10.670076+0000","last_active":"2024-03-26T21:48:36.903483+0000","last_peered":"2024-03-26T21:48:36.903483+0000","last_clean":"2024-03-26T21:48:36.903483+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:48:36.903483+0000","last_undegraded":"2024-03-26T21:48:36.903483+0000","last_fullsized":"2024-03-26T21:48:36.903483+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:10.669965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:47:10.669965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":870,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916055+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:48:41.916055+0000","last_peered":"2024-03-26T21:48:41.916055+0000","last_clean":"2024-03-26T21:48:41.916055+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:48:41.916055+0000","last_undegraded":"2024-03-26T21:48:41.916055+0000","last_fullsized":"2024-03-26T21:48:41.916055+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"456'5498","reported_seq":5553,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.023518+0000","last_change":"2024-03-26T21:47:39.047378+0000","last_active":"2024-03-26T21:48:43.023518+0000","last_peered":"2024-03-26T21:48:43.023518+0000","last_clean":"2024-03-26T21:48:43.023518+0000","last_became_active":"2024-03-26T21:47:39.047073+0000","last_became_peered":"2024-03-26T21:47:39.047073+0000","last_unstale":"2024-03-26T21:48:43.023518+0000","last_undegraded":"2024-03-26T21:48:43.023518+0000","last_fullsized":"2024-03-26T21:48:43.023518+0000","mapping_epoch":433,"log_start":"440'700","ondisk_log_start":"440'700","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.356820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":351872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":790,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914587+0000","last_change":"2024-03-26T21:48:02.882565+0000","last_active":"2024-03-26T21:48:41.914587+0000","last_peered":"2024-03-26T21:48:41.914587+0000","last_clean":"2024-03-26T21:48:41.914587+0000","last_became_active":"2024-03-26T21:41:25.608190+0000","last_became_peered":"2024-03-26T21:41:25.608190+0000","last_unstale":"2024-03-26T21:48:41.914587+0000","last_undegraded":"2024-03-26T21:48:41.914587+0000","last_fullsized":"2024-03-26T21:48:41.914587+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.981858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"277'42","reported_seq":1225,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.304623+0000","last_change":"2024-03-26T21:48:23.427353+0000","last_active":"2024-03-26T21:48:41.304623+0000","last_peered":"2024-03-26T21:48:41.304623+0000","last_clean":"2024-03-26T21:48:41.304623+0000","last_became_active":"2024-03-26T21:41:25.609228+0000","last_became_peered":"2024-03-26T21:41:25.609228+0000","last_unstale":"2024-03-26T21:48:41.304623+0000","last_undegraded":"2024-03-26T21:48:41.304623+0000","last_fullsized":"2024-03-26T21:48:41.304623+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.771806+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.926177+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:48:41.926177+0000","last_peered":"2024-03-26T21:48:41.926177+0000","last_clean":"2024-03-26T21:48:41.926177+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:48:41.926177+0000","last_undegraded":"2024-03-26T21:48:41.926177+0000","last_fullsized":"2024-03-26T21:48:41.926177+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1058,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914824+0000","last_change":"2024-03-26T21:48:27.821566+0000","last_active":"2024-03-26T21:48:41.914824+0000","last_peered":"2024-03-26T21:48:41.914824+0000","last_clean":"2024-03-26T21:48:41.914824+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:48:41.914824+0000","last_undegraded":"2024-03-26T21:48:41.914824+0000","last_fullsized":"2024-03-26T21:48:41.914824+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:48:27.821430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.821430+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:46.416532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":873,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910557+0000","last_change":"2024-03-26T21:48:05.886697+0000","last_active":"2024-03-26T21:48:41.910557+0000","last_peered":"2024-03-26T21:48:41.910557+0000","last_clean":"2024-03-26T21:48:41.910557+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:48:41.910557+0000","last_undegraded":"2024-03-26T21:48:41.910557+0000","last_fullsized":"2024-03-26T21:48:41.910557+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.886465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.886465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.802281+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"456'5442","reported_seq":5498,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.989090+0000","last_change":"2024-03-26T21:47:39.047146+0000","last_active":"2024-03-26T21:48:42.989090+0000","last_peered":"2024-03-26T21:48:42.989090+0000","last_clean":"2024-03-26T21:48:42.989090+0000","last_became_active":"2024-03-26T21:47:39.046837+0000","last_became_peered":"2024-03-26T21:47:39.046837+0000","last_unstale":"2024-03-26T21:48:42.989090+0000","last_undegraded":"2024-03-26T21:48:42.989090+0000","last_fullsized":"2024-03-26T21:48:42.989090+0000","mapping_epoch":433,"log_start":"440'600","ondisk_log_start":"440'600","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.520342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356646912,"num_objects":5442,"num_object_clones":0,"num_object_copies":16326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":348288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":871,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916754+0000","last_change":"2024-03-26T21:47:28.151192+0000","last_active":"2024-03-26T21:48:41.916754+0000","last_peered":"2024-03-26T21:48:41.916754+0000","last_clean":"2024-03-26T21:48:41.916754+0000","last_became_active":"2024-03-26T21:41:25.611068+0000","last_became_peered":"2024-03-26T21:41:25.611068+0000","last_unstale":"2024-03-26T21:48:41.916754+0000","last_undegraded":"2024-03-26T21:48:41.916754+0000","last_fullsized":"2024-03-26T21:48:41.916754+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.125869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.125869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:48.615066+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'24","reported_seq":915,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.324380+0000","last_change":"2024-03-26T21:48:35.767934+0000","last_active":"2024-03-26T21:48:42.324380+0000","last_peered":"2024-03-26T21:48:42.324380+0000","last_clean":"2024-03-26T21:48:42.324380+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:48:42.324380+0000","last_undegraded":"2024-03-26T21:48:42.324380+0000","last_fullsized":"2024-03-26T21:48:42.324380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:48:35.767034+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.767034+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:00.271391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":897,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915683+0000","last_change":"2024-03-26T21:48:17.180103+0000","last_active":"2024-03-26T21:48:41.915683+0000","last_peered":"2024-03-26T21:48:41.915683+0000","last_clean":"2024-03-26T21:48:41.915683+0000","last_became_active":"2024-03-26T21:44:49.410374+0000","last_became_peered":"2024-03-26T21:44:49.410374+0000","last_unstale":"2024-03-26T21:48:41.915683+0000","last_undegraded":"2024-03-26T21:48:41.915683+0000","last_fullsized":"2024-03-26T21:48:41.915683+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.339582+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":879,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916660+0000","last_change":"2024-03-26T21:48:05.696135+0000","last_active":"2024-03-26T21:48:41.916660+0000","last_peered":"2024-03-26T21:48:41.916660+0000","last_clean":"2024-03-26T21:48:41.916660+0000","last_became_active":"2024-03-26T21:28:34.851747+0000","last_became_peered":"2024-03-26T21:28:34.851747+0000","last_unstale":"2024-03-26T21:48:41.916660+0000","last_undegraded":"2024-03-26T21:48:41.916660+0000","last_fullsized":"2024-03-26T21:48:41.916660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:30.580793+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":778,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914903+0000","last_change":"2024-03-26T21:48:31.817091+0000","last_active":"2024-03-26T21:48:41.914903+0000","last_peered":"2024-03-26T21:48:41.914903+0000","last_clean":"2024-03-26T21:48:41.914903+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:48:41.914903+0000","last_undegraded":"2024-03-26T21:48:41.914903+0000","last_fullsized":"2024-03-26T21:48:41.914903+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:31.817002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.817002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:01.719710+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"456'5210","reported_seq":5266,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.992548+0000","last_change":"2024-03-26T21:47:39.051677+0000","last_active":"2024-03-26T21:48:42.992548+0000","last_peered":"2024-03-26T21:48:42.992548+0000","last_clean":"2024-03-26T21:48:42.992548+0000","last_became_active":"2024-03-26T21:47:39.051279+0000","last_became_peered":"2024-03-26T21:47:39.051279+0000","last_unstale":"2024-03-26T21:48:42.992548+0000","last_undegraded":"2024-03-26T21:48:42.992548+0000","last_fullsized":"2024-03-26T21:48:42.992548+0000","mapping_epoch":433,"log_start":"440'400","ondisk_log_start":"440'400","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341442560,"num_objects":5210,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5210,"num_write_kb":333440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":863,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.901915+0000","last_change":"2024-03-26T21:48:32.000228+0000","last_active":"2024-03-26T21:48:36.901915+0000","last_peered":"2024-03-26T21:48:36.901915+0000","last_clean":"2024-03-26T21:48:36.901915+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:48:36.901915+0000","last_undegraded":"2024-03-26T21:48:36.901915+0000","last_fullsized":"2024-03-26T21:48:36.901915+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.000079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:37.990181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"364'43","reported_seq":1175,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910249+0000","last_change":"2024-03-26T21:48:14.657443+0000","last_active":"2024-03-26T21:48:41.910249+0000","last_peered":"2024-03-26T21:48:41.910249+0000","last_clean":"2024-03-26T21:48:41.910249+0000","last_became_active":"2024-03-26T21:44:49.397229+0000","last_became_peered":"2024-03-26T21:44:49.397229+0000","last_unstale":"2024-03-26T21:48:41.910249+0000","last_undegraded":"2024-03-26T21:48:41.910249+0000","last_fullsized":"2024-03-26T21:48:41.910249+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'43","last_scrub_stamp":"2024-03-26T21:48:14.657283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.657283+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.781065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":296,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915699+0000","last_change":"2024-03-26T21:48:14.161052+0000","last_active":"2024-03-26T21:48:41.915699+0000","last_peered":"2024-03-26T21:48:41.915699+0000","last_clean":"2024-03-26T21:48:41.915699+0000","last_became_active":"2024-03-26T21:43:16.796261+0000","last_became_peered":"2024-03-26T21:43:16.796261+0000","last_unstale":"2024-03-26T21:48:41.915699+0000","last_undegraded":"2024-03-26T21:48:41.915699+0000","last_fullsized":"2024-03-26T21:48:41.915699+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:27.059211+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":870,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910532+0000","last_change":"2024-03-26T21:47:12.568606+0000","last_active":"2024-03-26T21:48:41.910532+0000","last_peered":"2024-03-26T21:48:41.910532+0000","last_clean":"2024-03-26T21:48:41.910532+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:48:41.910532+0000","last_undegraded":"2024-03-26T21:48:41.910532+0000","last_fullsized":"2024-03-26T21:48:41.910532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:47:12.568413+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:47:12.568413+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.568413+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927009+0000","last_change":"2024-03-26T21:48:29.771118+0000","last_active":"2024-03-26T21:48:41.927009+0000","last_peered":"2024-03-26T21:48:41.927009+0000","last_clean":"2024-03-26T21:48:41.927009+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:48:41.927009+0000","last_undegraded":"2024-03-26T21:48:41.927009+0000","last_fullsized":"2024-03-26T21:48:41.927009+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.770950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.770950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:38.482257+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"456'5415","reported_seq":5471,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.329661+0000","last_change":"2024-03-26T21:47:39.048784+0000","last_active":"2024-03-26T21:48:43.329661+0000","last_peered":"2024-03-26T21:48:43.329661+0000","last_clean":"2024-03-26T21:48:43.329661+0000","last_became_active":"2024-03-26T21:47:39.047884+0000","last_became_peered":"2024-03-26T21:47:39.047884+0000","last_unstale":"2024-03-26T21:48:43.329661+0000","last_undegraded":"2024-03-26T21:48:43.329661+0000","last_fullsized":"2024-03-26T21:48:43.329661+0000","mapping_epoch":433,"log_start":"440'300","ondisk_log_start":"440'300","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.003669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354877440,"num_objects":5415,"num_object_clones":0,"num_object_copies":16245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":346560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"234'15","reported_seq":895,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.906128+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:48:36.906128+0000","last_peered":"2024-03-26T21:48:36.906128+0000","last_clean":"2024-03-26T21:48:36.906128+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:48:36.906128+0000","last_undegraded":"2024-03-26T21:48:36.906128+0000","last_fullsized":"2024-03-26T21:48:36.906128+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:17.027797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":879,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910560+0000","last_change":"2024-03-26T21:48:23.858924+0000","last_active":"2024-03-26T21:48:41.910560+0000","last_peered":"2024-03-26T21:48:41.910560+0000","last_clean":"2024-03-26T21:48:41.910560+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:48:41.910560+0000","last_undegraded":"2024-03-26T21:48:41.910560+0000","last_fullsized":"2024-03-26T21:48:41.910560+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.858784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.858784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:25.054542+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":895,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915705+0000","last_change":"2024-03-26T21:48:04.064240+0000","last_active":"2024-03-26T21:48:41.915705+0000","last_peered":"2024-03-26T21:48:41.915705+0000","last_clean":"2024-03-26T21:48:41.915705+0000","last_became_active":"2024-03-26T21:44:49.410172+0000","last_became_peered":"2024-03-26T21:44:49.410172+0000","last_unstale":"2024-03-26T21:48:41.915705+0000","last_undegraded":"2024-03-26T21:48:41.915705+0000","last_fullsized":"2024-03-26T21:48:41.915705+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:09.204031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.903762+0000","last_change":"2024-03-26T21:48:03.013891+0000","last_active":"2024-03-26T21:48:36.903762+0000","last_peered":"2024-03-26T21:48:36.903762+0000","last_clean":"2024-03-26T21:48:36.903762+0000","last_became_active":"2024-03-26T21:41:25.611630+0000","last_became_peered":"2024-03-26T21:41:25.611630+0000","last_unstale":"2024-03-26T21:48:36.903762+0000","last_undegraded":"2024-03-26T21:48:36.903762+0000","last_fullsized":"2024-03-26T21:48:36.903762+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.674529+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"456'5137","reported_seq":5193,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:43.338061+0000","last_change":"2024-03-26T21:47:39.046775+0000","last_active":"2024-03-26T21:48:43.338061+0000","last_peered":"2024-03-26T21:48:43.338061+0000","last_clean":"2024-03-26T21:48:43.338061+0000","last_became_active":"2024-03-26T21:47:39.046369+0000","last_became_peered":"2024-03-26T21:47:39.046369+0000","last_unstale":"2024-03-26T21:48:43.338061+0000","last_undegraded":"2024-03-26T21:48:43.338061+0000","last_fullsized":"2024-03-26T21:48:43.338061+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:06.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336658432,"num_objects":5137,"num_object_clones":0,"num_object_copies":15411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5137,"num_write_kb":328768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"234'44","reported_seq":937,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.304135+0000","last_change":"2024-03-26T21:48:01.054496+0000","last_active":"2024-03-26T21:48:41.304135+0000","last_peered":"2024-03-26T21:48:41.304135+0000","last_clean":"2024-03-26T21:48:41.304135+0000","last_became_active":"2024-03-26T21:41:25.618127+0000","last_became_peered":"2024-03-26T21:41:25.618127+0000","last_unstale":"2024-03-26T21:48:41.304135+0000","last_undegraded":"2024-03-26T21:48:41.304135+0000","last_fullsized":"2024-03-26T21:48:41.304135+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:02.422288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":805,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915366+0000","last_change":"2024-03-26T21:47:28.835918+0000","last_active":"2024-03-26T21:48:41.915366+0000","last_peered":"2024-03-26T21:48:41.915366+0000","last_clean":"2024-03-26T21:48:41.915366+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:48:41.915366+0000","last_undegraded":"2024-03-26T21:48:41.915366+0000","last_fullsized":"2024-03-26T21:48:41.915366+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.835830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.835830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:42.287565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":876,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915611+0000","last_change":"2024-03-26T21:48:02.055651+0000","last_active":"2024-03-26T21:48:41.915611+0000","last_peered":"2024-03-26T21:48:41.915611+0000","last_clean":"2024-03-26T21:48:41.915611+0000","last_became_active":"2024-03-26T21:28:38.869079+0000","last_became_peered":"2024-03-26T21:28:38.869079+0000","last_unstale":"2024-03-26T21:48:41.915611+0000","last_undegraded":"2024-03-26T21:48:41.915611+0000","last_fullsized":"2024-03-26T21:48:41.915611+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:10.758127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":778,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.913836+0000","last_change":"2024-03-26T21:48:10.897127+0000","last_active":"2024-03-26T21:48:41.913836+0000","last_peered":"2024-03-26T21:48:41.913836+0000","last_clean":"2024-03-26T21:48:41.913836+0000","last_became_active":"2024-03-26T21:44:49.396166+0000","last_became_peered":"2024-03-26T21:44:49.396166+0000","last_unstale":"2024-03-26T21:48:41.913836+0000","last_undegraded":"2024-03-26T21:48:41.913836+0000","last_fullsized":"2024-03-26T21:48:41.913836+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:19.451684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"454'4912","reported_seq":4964,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:39.014995+0000","last_change":"2024-03-26T21:47:39.055017+0000","last_active":"2024-03-26T21:48:39.014995+0000","last_peered":"2024-03-26T21:48:39.014995+0000","last_clean":"2024-03-26T21:48:39.014995+0000","last_became_active":"2024-03-26T21:47:39.054705+0000","last_became_peered":"2024-03-26T21:47:39.054705+0000","last_unstale":"2024-03-26T21:48:39.014995+0000","last_undegraded":"2024-03-26T21:48:39.014995+0000","last_fullsized":"2024-03-26T21:48:39.014995+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:58.901128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":314368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":878,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915284+0000","last_change":"2024-03-26T21:48:23.129601+0000","last_active":"2024-03-26T21:48:41.915284+0000","last_peered":"2024-03-26T21:48:41.915284+0000","last_clean":"2024-03-26T21:48:41.915284+0000","last_became_active":"2024-03-26T21:28:38.866579+0000","last_became_peered":"2024-03-26T21:28:38.866579+0000","last_unstale":"2024-03-26T21:48:41.915284+0000","last_undegraded":"2024-03-26T21:48:41.915284+0000","last_fullsized":"2024-03-26T21:48:41.915284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.123676+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"282'23","reported_seq":608,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907781+0000","last_change":"2024-03-26T21:48:36.706426+0000","last_active":"2024-03-26T21:48:36.907781+0000","last_peered":"2024-03-26T21:48:36.907781+0000","last_clean":"2024-03-26T21:48:36.907781+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:48:36.907781+0000","last_undegraded":"2024-03-26T21:48:36.907781+0000","last_fullsized":"2024-03-26T21:48:36.907781+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:48:36.706278+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.562121+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":843,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915250+0000","last_change":"2024-03-26T21:48:35.727970+0000","last_active":"2024-03-26T21:48:41.915250+0000","last_peered":"2024-03-26T21:48:41.915250+0000","last_clean":"2024-03-26T21:48:41.915250+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:48:41.915250+0000","last_undegraded":"2024-03-26T21:48:41.915250+0000","last_fullsized":"2024-03-26T21:48:41.915250+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.727841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.817821+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927095+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:48:41.927095+0000","last_peered":"2024-03-26T21:48:41.927095+0000","last_clean":"2024-03-26T21:48:41.927095+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:48:41.927095+0000","last_undegraded":"2024-03-26T21:48:41.927095+0000","last_fullsized":"2024-03-26T21:48:41.927095+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.322569+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"456'5265","reported_seq":5319,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.127234+0000","last_change":"2024-03-26T21:47:39.051767+0000","last_active":"2024-03-26T21:48:42.127234+0000","last_peered":"2024-03-26T21:48:42.127234+0000","last_clean":"2024-03-26T21:48:42.127234+0000","last_became_active":"2024-03-26T21:47:39.051349+0000","last_became_peered":"2024-03-26T21:47:39.051349+0000","last_unstale":"2024-03-26T21:48:42.127234+0000","last_undegraded":"2024-03-26T21:48:42.127234+0000","last_fullsized":"2024-03-26T21:48:42.127234+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:55.642445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345047040,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5265,"num_write_kb":336960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1040,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.913877+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:48:41.913877+0000","last_peered":"2024-03-26T21:48:41.913877+0000","last_clean":"2024-03-26T21:48:41.913877+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:48:41.913877+0000","last_undegraded":"2024-03-26T21:48:41.913877+0000","last_fullsized":"2024-03-26T21:48:41.913877+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:59.821327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"234'27","reported_seq":908,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910092+0000","last_change":"2024-03-26T21:48:17.779307+0000","last_active":"2024-03-26T21:48:41.910092+0000","last_peered":"2024-03-26T21:48:41.910092+0000","last_clean":"2024-03-26T21:48:41.910092+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:48:41.910092+0000","last_undegraded":"2024-03-26T21:48:41.910092+0000","last_fullsized":"2024-03-26T21:48:41.910092+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:17.779045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.779045+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:33.002288+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":852,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915012+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:48:41.915012+0000","last_peered":"2024-03-26T21:48:41.915012+0000","last_clean":"2024-03-26T21:48:41.915012+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:48:41.915012+0000","last_undegraded":"2024-03-26T21:48:41.915012+0000","last_fullsized":"2024-03-26T21:48:41.915012+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:31.675781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":848,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910890+0000","last_change":"2024-03-26T21:48:27.000729+0000","last_active":"2024-03-26T21:48:41.910890+0000","last_peered":"2024-03-26T21:48:41.910890+0000","last_clean":"2024-03-26T21:48:41.910890+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:48:41.910890+0000","last_undegraded":"2024-03-26T21:48:41.910890+0000","last_fullsized":"2024-03-26T21:48:41.910890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:27.000615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.000615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.371457+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'14","reported_seq":215,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.210845+0000","last_change":"2024-03-26T21:47:59.714612+0000","last_active":"2024-03-26T21:48:42.210845+0000","last_peered":"2024-03-26T21:48:42.210845+0000","last_clean":"2024-03-26T21:48:42.210845+0000","last_became_active":"2024-03-26T21:44:49.408457+0000","last_became_peered":"2024-03-26T21:44:49.408457+0000","last_unstale":"2024-03-26T21:48:42.210845+0000","last_undegraded":"2024-03-26T21:48:42.210845+0000","last_fullsized":"2024-03-26T21:48:42.210845+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:06.709447+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":17,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":875,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.303518+0000","last_change":"2024-03-26T21:48:37.292092+0000","last_active":"2024-03-26T21:48:41.303518+0000","last_peered":"2024-03-26T21:48:41.303518+0000","last_clean":"2024-03-26T21:48:41.303518+0000","last_became_active":"2024-03-26T21:28:34.843728+0000","last_became_peered":"2024-03-26T21:28:34.843728+0000","last_unstale":"2024-03-26T21:48:41.303518+0000","last_undegraded":"2024-03-26T21:48:41.303518+0000","last_fullsized":"2024-03-26T21:48:41.303518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.291965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:42.740930+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":863,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910883+0000","last_change":"2024-03-26T21:48:07.714924+0000","last_active":"2024-03-26T21:48:41.910883+0000","last_peered":"2024-03-26T21:48:41.910883+0000","last_clean":"2024-03-26T21:48:41.910883+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:48:41.910883+0000","last_undegraded":"2024-03-26T21:48:41.910883+0000","last_fullsized":"2024-03-26T21:48:41.910883+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.714744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.714744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:33.078685+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":788,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914953+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:48:41.914953+0000","last_peered":"2024-03-26T21:48:41.914953+0000","last_clean":"2024-03-26T21:48:41.914953+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:48:41.914953+0000","last_undegraded":"2024-03-26T21:48:41.914953+0000","last_fullsized":"2024-03-26T21:48:41.914953+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:38.657067+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"234'19","reported_seq":905,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.903042+0000","last_change":"2024-03-26T21:47:11.703126+0000","last_active":"2024-03-26T21:48:36.903042+0000","last_peered":"2024-03-26T21:48:36.903042+0000","last_clean":"2024-03-26T21:48:36.903042+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:48:36.903042+0000","last_undegraded":"2024-03-26T21:48:36.903042+0000","last_fullsized":"2024-03-26T21:48:36.903042+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:47:11.703039+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.703039+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":825,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914097+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:48:41.914097+0000","last_peered":"2024-03-26T21:48:41.914097+0000","last_clean":"2024-03-26T21:48:41.914097+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:48:41.914097+0000","last_undegraded":"2024-03-26T21:48:41.914097+0000","last_fullsized":"2024-03-26T21:48:41.914097+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:11.797721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.928321+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:48:41.928321+0000","last_peered":"2024-03-26T21:48:41.928321+0000","last_clean":"2024-03-26T21:48:41.928321+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:48:41.928321+0000","last_undegraded":"2024-03-26T21:48:41.928321+0000","last_fullsized":"2024-03-26T21:48:41.928321+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.275926+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":879,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916671+0000","last_change":"2024-03-26T21:47:18.904009+0000","last_active":"2024-03-26T21:48:41.916671+0000","last_peered":"2024-03-26T21:48:41.916671+0000","last_clean":"2024-03-26T21:48:41.916671+0000","last_became_active":"2024-03-26T21:41:25.600955+0000","last_became_peered":"2024-03-26T21:41:25.600955+0000","last_unstale":"2024-03-26T21:48:41.916671+0000","last_undegraded":"2024-03-26T21:48:41.916671+0000","last_fullsized":"2024-03-26T21:48:41.916671+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:45.184979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"234'19","reported_seq":923,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910838+0000","last_change":"2024-03-26T21:48:20.776635+0000","last_active":"2024-03-26T21:48:41.910838+0000","last_peered":"2024-03-26T21:48:41.910838+0000","last_clean":"2024-03-26T21:48:41.910838+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:48:41.910838+0000","last_undegraded":"2024-03-26T21:48:41.910838+0000","last_fullsized":"2024-03-26T21:48:41.910838+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:20.776494+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.776494+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.363911+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":860,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907135+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:48:36.907135+0000","last_peered":"2024-03-26T21:48:36.907135+0000","last_clean":"2024-03-26T21:48:36.907135+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:48:36.907135+0000","last_undegraded":"2024-03-26T21:48:36.907135+0000","last_fullsized":"2024-03-26T21:48:36.907135+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:37.946686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"364'13","reported_seq":255,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.928127+0000","last_change":"2024-03-26T21:48:33.859254+0000","last_active":"2024-03-26T21:48:41.928127+0000","last_peered":"2024-03-26T21:48:41.928127+0000","last_clean":"2024-03-26T21:48:41.928127+0000","last_became_active":"2024-03-26T21:44:49.402177+0000","last_became_peered":"2024-03-26T21:44:49.402177+0000","last_unstale":"2024-03-26T21:48:41.928127+0000","last_undegraded":"2024-03-26T21:48:41.928127+0000","last_fullsized":"2024-03-26T21:48:41.928127+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13","last_scrub_stamp":"2024-03-26T21:48:33.859117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.859117+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:03.014109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":544,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.908325+0000","last_change":"2024-03-26T21:47:58.705838+0000","last_active":"2024-03-26T21:48:36.908325+0000","last_peered":"2024-03-26T21:48:36.908325+0000","last_clean":"2024-03-26T21:48:36.908325+0000","last_became_active":"2024-03-26T21:36:51.674084+0000","last_became_peered":"2024-03-26T21:36:51.674084+0000","last_unstale":"2024-03-26T21:48:36.908325+0000","last_undegraded":"2024-03-26T21:48:36.908325+0000","last_fullsized":"2024-03-26T21:48:36.908325+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:00.410661+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'27","reported_seq":219,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.223644+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:48:42.223644+0000","last_peered":"2024-03-26T21:48:42.223644+0000","last_clean":"2024-03-26T21:48:42.223644+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:48:42.223644+0000","last_undegraded":"2024-03-26T21:48:42.223644+0000","last_fullsized":"2024-03-26T21:48:42.223644+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:19.599750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":798,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.917967+0000","last_change":"2024-03-26T21:48:08.932035+0000","last_active":"2024-03-26T21:48:41.917967+0000","last_peered":"2024-03-26T21:48:41.917967+0000","last_clean":"2024-03-26T21:48:41.917967+0000","last_became_active":"2024-03-26T21:41:25.594520+0000","last_became_peered":"2024-03-26T21:41:25.594520+0000","last_unstale":"2024-03-26T21:48:41.917967+0000","last_undegraded":"2024-03-26T21:48:41.917967+0000","last_fullsized":"2024-03-26T21:48:41.917967+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:10.831931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1102,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910462+0000","last_change":"2024-03-26T21:48:41.813232+0000","last_active":"2024-03-26T21:48:41.910462+0000","last_peered":"2024-03-26T21:48:41.910462+0000","last_clean":"2024-03-26T21:48:41.910462+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:48:41.910462+0000","last_undegraded":"2024-03-26T21:48:41.910462+0000","last_fullsized":"2024-03-26T21:48:41.910462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:48:41.813092+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.813092+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.588630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":857,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902682+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:48:36.902682+0000","last_peered":"2024-03-26T21:48:36.902682+0000","last_clean":"2024-03-26T21:48:36.902682+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:48:36.902682+0000","last_undegraded":"2024-03-26T21:48:36.902682+0000","last_fullsized":"2024-03-26T21:48:36.902682+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:46.622715+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"424'24","reported_seq":575,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907211+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:48:36.907211+0000","last_peered":"2024-03-26T21:48:36.907211+0000","last_clean":"2024-03-26T21:48:36.907211+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:48:36.907211+0000","last_undegraded":"2024-03-26T21:48:36.907211+0000","last_fullsized":"2024-03-26T21:48:36.907211+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":555,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:39.735246+0000","last_change":"2024-03-26T21:48:39.735268+0000","last_active":"2024-03-26T21:48:39.735246+0000","last_peered":"2024-03-26T21:48:39.735246+0000","last_clean":"2024-03-26T21:48:39.735246+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:48:39.735246+0000","last_undegraded":"2024-03-26T21:48:39.735246+0000","last_fullsized":"2024-03-26T21:48:39.735246+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:48:39.735146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.746887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":885,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915741+0000","last_change":"2024-03-26T21:48:24.700288+0000","last_active":"2024-03-26T21:48:41.915741+0000","last_peered":"2024-03-26T21:48:41.915741+0000","last_clean":"2024-03-26T21:48:41.915741+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:48:41.915741+0000","last_undegraded":"2024-03-26T21:48:41.915741+0000","last_fullsized":"2024-03-26T21:48:41.915741+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:48:24.700160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.327860+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927504+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:48:41.927504+0000","last_peered":"2024-03-26T21:48:41.927504+0000","last_clean":"2024-03-26T21:48:41.927504+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:48:41.927504+0000","last_undegraded":"2024-03-26T21:48:41.927504+0000","last_fullsized":"2024-03-26T21:48:41.927504+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"234'19","reported_seq":894,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915944+0000","last_change":"2024-03-26T21:48:30.679273+0000","last_active":"2024-03-26T21:48:41.915944+0000","last_peered":"2024-03-26T21:48:41.915944+0000","last_clean":"2024-03-26T21:48:41.915944+0000","last_became_active":"2024-03-26T21:28:36.840692+0000","last_became_peered":"2024-03-26T21:28:36.840692+0000","last_unstale":"2024-03-26T21:48:41.915944+0000","last_undegraded":"2024-03-26T21:48:41.915944+0000","last_fullsized":"2024-03-26T21:48:41.915944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:30.679017+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.679017+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.141609+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":803,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915133+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:48:41.915133+0000","last_peered":"2024-03-26T21:48:41.915133+0000","last_clean":"2024-03-26T21:48:41.915133+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:48:41.915133+0000","last_undegraded":"2024-03-26T21:48:41.915133+0000","last_fullsized":"2024-03-26T21:48:41.915133+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.032406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.926880+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:48:41.926880+0000","last_peered":"2024-03-26T21:48:41.926880+0000","last_clean":"2024-03-26T21:48:41.926880+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:48:41.926880+0000","last_undegraded":"2024-03-26T21:48:41.926880+0000","last_fullsized":"2024-03-26T21:48:41.926880+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.201839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":837,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.901117+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:48:36.901117+0000","last_peered":"2024-03-26T21:48:36.901117+0000","last_clean":"2024-03-26T21:48:36.901117+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:48:36.901117+0000","last_undegraded":"2024-03-26T21:48:36.901117+0000","last_fullsized":"2024-03-26T21:48:36.901117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:13.726112+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"234'19","reported_seq":224,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.924511+0000","last_change":"2024-03-26T21:48:38.975837+0000","last_active":"2024-03-26T21:48:41.924511+0000","last_peered":"2024-03-26T21:48:41.924511+0000","last_clean":"2024-03-26T21:48:41.924511+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:48:41.924511+0000","last_undegraded":"2024-03-26T21:48:41.924511+0000","last_fullsized":"2024-03-26T21:48:41.924511+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:38.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:38.975710+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:05.250119+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":868,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910637+0000","last_change":"2024-03-26T21:48:12.733553+0000","last_active":"2024-03-26T21:48:41.910637+0000","last_peered":"2024-03-26T21:48:41.910637+0000","last_clean":"2024-03-26T21:48:41.910637+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:48:41.910637+0000","last_undegraded":"2024-03-26T21:48:41.910637+0000","last_fullsized":"2024-03-26T21:48:41.910637+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.733428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.733428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.412196+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":780,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914702+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:48:41.914702+0000","last_peered":"2024-03-26T21:48:41.914702+0000","last_clean":"2024-03-26T21:48:41.914702+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:48:41.914702+0000","last_undegraded":"2024-03-26T21:48:41.914702+0000","last_fullsized":"2024-03-26T21:48:41.914702+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.855255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":212,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.926990+0000","last_change":"2024-03-26T21:48:37.104233+0000","last_active":"2024-03-26T21:48:41.926990+0000","last_peered":"2024-03-26T21:48:41.926990+0000","last_clean":"2024-03-26T21:48:41.926990+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:48:41.926990+0000","last_undegraded":"2024-03-26T21:48:41.926990+0000","last_fullsized":"2024-03-26T21:48:41.926990+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.104108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.104108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.248220+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"234'8","reported_seq":218,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.913606+0000","last_change":"2024-03-26T21:48:42.913651+0000","last_active":"2024-03-26T21:48:42.913606+0000","last_peered":"2024-03-26T21:48:42.913606+0000","last_clean":"2024-03-26T21:48:42.913606+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:48:42.913606+0000","last_undegraded":"2024-03-26T21:48:42.913606+0000","last_fullsized":"2024-03-26T21:48:42.913606+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:48:42.913438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:42.913438+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:46.025071+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":861,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902333+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:48:36.902333+0000","last_peered":"2024-03-26T21:48:36.902333+0000","last_clean":"2024-03-26T21:48:36.902333+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:48:36.902333+0000","last_undegraded":"2024-03-26T21:48:36.902333+0000","last_fullsized":"2024-03-26T21:48:36.902333+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:26.599241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":889,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.305531+0000","last_change":"2024-03-26T21:48:26.476675+0000","last_active":"2024-03-26T21:48:41.305531+0000","last_peered":"2024-03-26T21:48:41.305531+0000","last_clean":"2024-03-26T21:48:41.305531+0000","last_became_active":"2024-03-26T21:41:25.609258+0000","last_became_peered":"2024-03-26T21:41:25.609258+0000","last_unstale":"2024-03-26T21:48:41.305531+0000","last_undegraded":"2024-03-26T21:48:41.305531+0000","last_fullsized":"2024-03-26T21:48:41.305531+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.591969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":788,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.913805+0000","last_change":"2024-03-26T21:48:22.860635+0000","last_active":"2024-03-26T21:48:41.913805+0000","last_peered":"2024-03-26T21:48:41.913805+0000","last_clean":"2024-03-26T21:48:41.913805+0000","last_became_active":"2024-03-26T21:41:25.595824+0000","last_became_peered":"2024-03-26T21:41:25.595824+0000","last_unstale":"2024-03-26T21:48:41.913805+0000","last_undegraded":"2024-03-26T21:48:41.913805+0000","last_fullsized":"2024-03-26T21:48:41.913805+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.867613+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"234'24","reported_seq":908,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.304421+0000","last_change":"2024-03-26T21:48:31.342834+0000","last_active":"2024-03-26T21:48:41.304421+0000","last_peered":"2024-03-26T21:48:41.304421+0000","last_clean":"2024-03-26T21:48:41.304421+0000","last_became_active":"2024-03-26T21:28:36.844689+0000","last_became_peered":"2024-03-26T21:28:36.844689+0000","last_unstale":"2024-03-26T21:48:41.304421+0000","last_undegraded":"2024-03-26T21:48:41.304421+0000","last_fullsized":"2024-03-26T21:48:41.304421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:31.342662+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.342662+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:00.138638+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.924070+0000","last_change":"2024-03-26T21:48:26.761549+0000","last_active":"2024-03-26T21:48:41.924070+0000","last_peered":"2024-03-26T21:48:41.924070+0000","last_clean":"2024-03-26T21:48:41.924070+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:48:41.924070+0000","last_undegraded":"2024-03-26T21:48:41.924070+0000","last_fullsized":"2024-03-26T21:48:41.924070+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.760812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.760812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:40.969521+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.903185+0000","last_change":"2024-03-26T21:47:22.710800+0000","last_active":"2024-03-26T21:48:36.903185+0000","last_peered":"2024-03-26T21:48:36.903185+0000","last_clean":"2024-03-26T21:48:36.903185+0000","last_became_active":"2024-03-26T21:41:25.608869+0000","last_became_peered":"2024-03-26T21:41:25.608869+0000","last_unstale":"2024-03-26T21:48:36.903185+0000","last_undegraded":"2024-03-26T21:48:36.903185+0000","last_fullsized":"2024-03-26T21:48:36.903185+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:22.710723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:28.707018+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.710723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":877,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916096+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:48:41.916096+0000","last_peered":"2024-03-26T21:48:41.916096+0000","last_clean":"2024-03-26T21:48:41.916096+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:48:41.916096+0000","last_undegraded":"2024-03-26T21:48:41.916096+0000","last_fullsized":"2024-03-26T21:48:41.916096+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.137319+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"234'20","reported_seq":929,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902948+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:48:36.902948+0000","last_peered":"2024-03-26T21:48:36.902948+0000","last_clean":"2024-03-26T21:48:36.902948+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:48:36.902948+0000","last_undegraded":"2024-03-26T21:48:36.902948+0000","last_fullsized":"2024-03-26T21:48:36.902948+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.500276+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":912,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916066+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:48:41.916066+0000","last_peered":"2024-03-26T21:48:41.916066+0000","last_clean":"2024-03-26T21:48:41.916066+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:48:41.916066+0000","last_undegraded":"2024-03-26T21:48:41.916066+0000","last_fullsized":"2024-03-26T21:48:41.916066+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:16.571019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"277'14","reported_seq":1156,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.304410+0000","last_change":"2024-03-26T21:48:32.328080+0000","last_active":"2024-03-26T21:48:41.304410+0000","last_peered":"2024-03-26T21:48:41.304410+0000","last_clean":"2024-03-26T21:48:41.304410+0000","last_became_active":"2024-03-26T21:41:25.604989+0000","last_became_peered":"2024-03-26T21:41:25.604989+0000","last_unstale":"2024-03-26T21:48:41.304410+0000","last_undegraded":"2024-03-26T21:48:41.304410+0000","last_fullsized":"2024-03-26T21:48:41.304410+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:48:32.327875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.327875+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:52.241058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":783,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914775+0000","last_change":"2024-03-26T21:48:04.881122+0000","last_active":"2024-03-26T21:48:41.914775+0000","last_peered":"2024-03-26T21:48:41.914775+0000","last_clean":"2024-03-26T21:48:41.914775+0000","last_became_active":"2024-03-26T21:41:25.608163+0000","last_became_peered":"2024-03-26T21:41:25.608163+0000","last_unstale":"2024-03-26T21:48:41.914775+0000","last_undegraded":"2024-03-26T21:48:41.914775+0000","last_fullsized":"2024-03-26T21:48:41.914775+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:12.034875+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'23","reported_seq":221,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.924412+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:48:41.924412+0000","last_peered":"2024-03-26T21:48:41.924412+0000","last_clean":"2024-03-26T21:48:41.924412+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:48:41.924412+0000","last_undegraded":"2024-03-26T21:48:41.924412+0000","last_fullsized":"2024-03-26T21:48:41.924412+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907344+0000","last_change":"2024-03-26T21:48:07.663645+0000","last_active":"2024-03-26T21:48:36.907344+0000","last_peered":"2024-03-26T21:48:36.907344+0000","last_clean":"2024-03-26T21:48:36.907344+0000","last_became_active":"2024-03-26T21:36:51.696806+0000","last_became_peered":"2024-03-26T21:36:51.696806+0000","last_unstale":"2024-03-26T21:48:36.907344+0000","last_undegraded":"2024-03-26T21:48:36.907344+0000","last_fullsized":"2024-03-26T21:48:36.907344+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:37.361646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":569,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907695+0000","last_change":"2024-03-26T21:48:16.766872+0000","last_active":"2024-03-26T21:48:36.907695+0000","last_peered":"2024-03-26T21:48:36.907695+0000","last_clean":"2024-03-26T21:48:36.907695+0000","last_became_active":"2024-03-26T21:44:49.395939+0000","last_became_peered":"2024-03-26T21:44:49.395939+0000","last_unstale":"2024-03-26T21:48:36.907695+0000","last_undegraded":"2024-03-26T21:48:36.907695+0000","last_fullsized":"2024-03-26T21:48:36.907695+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.981952+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.927131+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:48:41.927131+0000","last_peered":"2024-03-26T21:48:41.927131+0000","last_clean":"2024-03-26T21:48:41.927131+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:48:41.927131+0000","last_undegraded":"2024-03-26T21:48:41.927131+0000","last_fullsized":"2024-03-26T21:48:41.927131+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:26.679721+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"234'27","reported_seq":960,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916451+0000","last_change":"2024-03-26T21:48:09.176526+0000","last_active":"2024-03-26T21:48:41.916451+0000","last_peered":"2024-03-26T21:48:41.916451+0000","last_clean":"2024-03-26T21:48:41.916451+0000","last_became_active":"2024-03-26T21:44:49.417143+0000","last_became_peered":"2024-03-26T21:44:49.417143+0000","last_unstale":"2024-03-26T21:48:41.916451+0000","last_undegraded":"2024-03-26T21:48:41.916451+0000","last_fullsized":"2024-03-26T21:48:41.916451+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:25.793568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":863,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.905920+0000","last_change":"2024-03-26T21:47:12.688818+0000","last_active":"2024-03-26T21:48:36.905920+0000","last_peered":"2024-03-26T21:48:36.905920+0000","last_clean":"2024-03-26T21:48:36.905920+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:48:36.905920+0000","last_undegraded":"2024-03-26T21:48:36.905920+0000","last_fullsized":"2024-03-26T21:48:36.905920+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.688726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.688726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":874,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.916518+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:48:41.916518+0000","last_peered":"2024-03-26T21:48:41.916518+0000","last_clean":"2024-03-26T21:48:41.916518+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:48:41.916518+0000","last_undegraded":"2024-03-26T21:48:41.916518+0000","last_fullsized":"2024-03-26T21:48:41.916518+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:28.449043+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":546,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907928+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:48:36.907928+0000","last_peered":"2024-03-26T21:48:36.907928+0000","last_clean":"2024-03-26T21:48:36.907928+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:48:36.907928+0000","last_undegraded":"2024-03-26T21:48:36.907928+0000","last_fullsized":"2024-03-26T21:48:36.907928+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:43.666911+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"234'7","reported_seq":578,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907312+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:48:36.907312+0000","last_peered":"2024-03-26T21:48:36.907312+0000","last_clean":"2024-03-26T21:48:36.907312+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:48:36.907312+0000","last_undegraded":"2024-03-26T21:48:36.907312+0000","last_fullsized":"2024-03-26T21:48:36.907312+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.179909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":880,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910523+0000","last_change":"2024-03-26T21:48:29.849767+0000","last_active":"2024-03-26T21:48:41.910523+0000","last_peered":"2024-03-26T21:48:41.910523+0000","last_clean":"2024-03-26T21:48:41.910523+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:48:41.910523+0000","last_undegraded":"2024-03-26T21:48:41.910523+0000","last_fullsized":"2024-03-26T21:48:41.910523+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.849687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:58.749020+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":519,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.914398+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:48:41.914398+0000","last_peered":"2024-03-26T21:48:41.914398+0000","last_clean":"2024-03-26T21:48:41.914398+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:48:41.914398+0000","last_undegraded":"2024-03-26T21:48:41.914398+0000","last_fullsized":"2024-03-26T21:48:41.914398+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:08.966616+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":859,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910155+0000","last_change":"2024-03-26T21:48:03.850832+0000","last_active":"2024-03-26T21:48:41.910155+0000","last_peered":"2024-03-26T21:48:41.910155+0000","last_clean":"2024-03-26T21:48:41.910155+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:48:41.910155+0000","last_undegraded":"2024-03-26T21:48:41.910155+0000","last_fullsized":"2024-03-26T21:48:41.910155+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.850650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.850650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:32.318466+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"234'19","reported_seq":604,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907558+0000","last_change":"2024-03-26T21:48:27.673802+0000","last_active":"2024-03-26T21:48:36.907558+0000","last_peered":"2024-03-26T21:48:36.907558+0000","last_clean":"2024-03-26T21:48:36.907558+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:48:36.907558+0000","last_undegraded":"2024-03-26T21:48:36.907558+0000","last_fullsized":"2024-03-26T21:48:36.907558+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:27.673516+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.673516+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:55.239856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":880,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910368+0000","last_change":"2024-03-26T21:48:34.769285+0000","last_active":"2024-03-26T21:48:41.910368+0000","last_peered":"2024-03-26T21:48:41.910368+0000","last_clean":"2024-03-26T21:48:41.910368+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:48:41.910368+0000","last_undegraded":"2024-03-26T21:48:41.910368+0000","last_fullsized":"2024-03-26T21:48:41.910368+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:34.769168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.769168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:37.772018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":860,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.910793+0000","last_change":"2024-03-26T21:48:30.838758+0000","last_active":"2024-03-26T21:48:41.910793+0000","last_peered":"2024-03-26T21:48:41.910793+0000","last_clean":"2024-03-26T21:48:41.910793+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:48:41.910793+0000","last_undegraded":"2024-03-26T21:48:41.910793+0000","last_fullsized":"2024-03-26T21:48:41.910793+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.838637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:30.838637+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.838637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:57.825792+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":838,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.901407+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:48:36.901407+0000","last_peered":"2024-03-26T21:48:36.901407+0000","last_clean":"2024-03-26T21:48:36.901407+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:48:36.901407+0000","last_undegraded":"2024-03-26T21:48:36.901407+0000","last_fullsized":"2024-03-26T21:48:36.901407+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:22.767114+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'29","reported_seq":223,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:42.989948+0000","last_change":"2024-03-26T21:48:34.910615+0000","last_active":"2024-03-26T21:48:42.989948+0000","last_peered":"2024-03-26T21:48:42.989948+0000","last_clean":"2024-03-26T21:48:42.989948+0000","last_became_active":"2024-03-26T21:44:49.407818+0000","last_became_peered":"2024-03-26T21:44:49.407818+0000","last_unstale":"2024-03-26T21:48:42.989948+0000","last_undegraded":"2024-03-26T21:48:42.989948+0000","last_fullsized":"2024-03-26T21:48:42.989948+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:34.910272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.910272+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.060233+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.926921+0000","last_change":"2024-03-26T21:47:58.673734+0000","last_active":"2024-03-26T21:48:41.926921+0000","last_peered":"2024-03-26T21:48:41.926921+0000","last_clean":"2024-03-26T21:48:41.926921+0000","last_became_active":"2024-03-26T21:44:49.404120+0000","last_became_peered":"2024-03-26T21:44:49.404120+0000","last_unstale":"2024-03-26T21:48:41.926921+0000","last_undegraded":"2024-03-26T21:48:41.926921+0000","last_fullsized":"2024-03-26T21:48:41.926921+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:23.724887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.906820+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:48:36.906820+0000","last_peered":"2024-03-26T21:48:36.906820+0000","last_clean":"2024-03-26T21:48:36.906820+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:48:36.906820+0000","last_undegraded":"2024-03-26T21:48:36.906820+0000","last_fullsized":"2024-03-26T21:48:36.906820+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:42.313798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":204,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.926765+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:48:41.926765+0000","last_peered":"2024-03-26T21:48:41.926765+0000","last_clean":"2024-03-26T21:48:41.926765+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:48:41.926765+0000","last_undegraded":"2024-03-26T21:48:41.926765+0000","last_fullsized":"2024-03-26T21:48:41.926765+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:48:52.864285+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"234'15","reported_seq":580,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.906916+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:48:36.906916+0000","last_peered":"2024-03-26T21:48:36.906916+0000","last_clean":"2024-03-26T21:48:36.906916+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:48:36.906916+0000","last_undegraded":"2024-03-26T21:48:36.906916+0000","last_fullsized":"2024-03-26T21:48:36.906916+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:24.532532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":850,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.902139+0000","last_change":"2024-03-26T21:48:33.984623+0000","last_active":"2024-03-26T21:48:36.902139+0000","last_peered":"2024-03-26T21:48:36.902139+0000","last_clean":"2024-03-26T21:48:36.902139+0000","last_became_active":"2024-03-26T21:28:34.844089+0000","last_became_peered":"2024-03-26T21:28:34.844089+0000","last_unstale":"2024-03-26T21:48:36.902139+0000","last_undegraded":"2024-03-26T21:48:36.902139+0000","last_fullsized":"2024-03-26T21:48:36.902139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:33.984501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.984501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:59.758828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":862,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-26T21:48:41.915812+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:48:41.915812+0000","last_peered":"2024-03-26T21:48:41.915812+0000","last_clean":"2024-03-26T21:48:41.915812+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:48:41.915812+0000","last_undegraded":"2024-03-26T21:48:41.915812+0000","last_fullsized":"2024-03-26T21:48:41.915812+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T21:48:36.907443+0000","last_change":"2024-03-26T21:47:17.587843+0000","last_active":"2024-03-26T21:48:36.907443+0000","last_peered":"2024-03-26T21:48:36.907443+0000","last_clean":"2024-03-26T21:48:36.907443+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:48:36.907443+0000","last_undegraded":"2024-03-26T21:48:36.907443+0000","last_fullsized":"2024-03-26T21:48:36.907443+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:17.587721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:17.587721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5513412608,"num_objects":84128,"num_object_clones":0,"num_object_copies":252384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84128,"num_write_kb":5384192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16483352576,"data_stored":16483352576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80628,"ondisk_log_size":80628,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1087,"num_read_kb":1082,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1136,"num_read_kb":893,"num_write":719,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973785,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2460852,"kb_used_data":2014436,"kb_used_omap":6,"kb_used_meta":446393,"kb_avail":91284300,"statfs":{"total":95995035648,"available":93475123200,"internally_reserved":0,"allocated":2062782464,"data_stored":2062101403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":457106728},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365052,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3310148,"kb_used_data":2661176,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":90435004,"statfs":{"total":95995035648,"available":92605444096,"internally_reserved":0,"allocated":2725044224,"data_stored":2724398925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3521408,"kb_used_data":2691952,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":90223744,"statfs":{"total":95995035648,"available":92389113856,"internally_reserved":0,"allocated":2756558848,"data_stored":2755864296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2370200,"kb_used_data":1594896,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":91374952,"statfs":{"total":95995035648,"available":93567950848,"internally_reserved":0,"allocated":1633173504,"data_stored":1632498525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411600,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2692232,"kb_used_data":2007164,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91052920,"statfs":{"total":95995035648,"available":93238190080,"internally_reserved":0,"allocated":2055335936,"data_stored":2054692994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149962,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3284664,"kb_used_data":2387176,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":90460488,"statfs":{"total":95995035648,"available":92631539712,"internally_reserved":0,"allocated":2444468224,"data_stored":2443781015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":919009735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650558,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2019732,"kb_used_data":1315660,"kb_used_omap":4,"kb_used_meta":704059,"kb_avail":91725420,"statfs":{"total":95995035648,"available":93926830080,"internally_reserved":0,"allocated":1347235840,"data_stored":1346564537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":720957091},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659391,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042132,"kb_used_data":1347336,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91703020,"statfs":{"total":95995035648,"available":93903892480,"internally_reserved":0,"allocated":1379672064,"data_stored":1379010521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1631518720,"data_stored":1631518720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2767716352,"data_stored":2767716352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2071920640,"data_stored":2071920640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2446458880,"data_stored":2446458880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1355087872,"data_stored":1355087872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1385431040,"data_stored":1385431040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2743861248,"data_stored":2743861248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2081357824,"data_stored":2081357824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:48:49.047 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T21:48:49.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:48:49.047 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 5, 6, 7, 4, 3] 2024-03-26T21:48:49.047 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd out 2 2024-03-26T21:48:49.057 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:49.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:49.206 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:49.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:49.334 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:49.335 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:48:49.335 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:48:49.335 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.335 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91242 11.2 11:48877cd2:::benchmark_data_smithi055_186849_object91241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.335 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.310695+0000", 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.029592842000000001, 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.029632962999999998, 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91242 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.336 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310695+0000", 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310695+0000", 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.337 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310695+0000", 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.63e-07 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310717+0000", 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1590000000000002e-05 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.338 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310724+0000", 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2880000000000004e-06 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310731+0000", 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.302e-06 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.320059+0000", 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0093282710000000008 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.339 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.320463+0000", 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00040381999999999999 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.320612+0000", 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014930799999999999 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.340 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91243 11.2 11:47bf01bc:::benchmark_data_smithi055_186849_object91242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.310739+0000", 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.029548828999999999, 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.029695783999999999, 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.341 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91243 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310739+0000", 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310739+0000", 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.342 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310739+0000", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0400000000000002e-07 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310758+0000", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8505e-05 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310764+0000", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6949999999999997e-06 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.343 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310769+0000", 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3240000000000002e-06 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.328017+0000", 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.017248023000000001 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.344 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.328159+0000", 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014177699999999999 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.328305+0000", 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000146563 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91254 11.2 11:448ac467:::benchmark_data_smithi055_186849_object91253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.323648+0000", 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.016639655, 2024-03-26T21:48:49.345 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016884585000000001, 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91254 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323648+0000", 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.346 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323648+0000", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323649+0000", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4600000000000004e-07 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323670+0000", 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1423000000000001e-05 2024-03-26T21:48:49.347 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.348 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.348 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.348 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323677+0000", 2024-03-26T21:48:49.348 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3200000000000002e-06 2024-03-26T21:48:49.348 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.323684+0000", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.528e-06 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331576+0000", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0078916160000000006 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331878+0000", 2024-03-26T21:48:49.349 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030195800000000001 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.332367+0000", 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00048933899999999998 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91256 11.2 11:4c338350:::benchmark_data_smithi055_186849_object91255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.325253+0000", 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.015034817000000001, 2024-03-26T21:48:49.350 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.015375461, 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91256 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325253+0000", 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.351 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325253+0000", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325254+0000", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.108e-06 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325277+0000", 2024-03-26T21:48:49.352 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3056000000000001e-05 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325295+0000", 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7515000000000001e-05 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.325303+0000", 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6379999999999993e-06 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.353 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338591+0000", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.013287874999999999 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338947+0000", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000355671 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339416+0000", 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00046947900000000002 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.354 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91258 11.2 11.db32652 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.329903+0000", 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.010384302999999999, 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010812769, 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "queued for pg", 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.355 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91258 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329903+0000", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329903+0000", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.356 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329905+0000", 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.601e-06 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329929+0000", 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3632999999999999e-05 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.357 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.358 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329945+0000", 2024-03-26T21:48:49.358 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.596e-05 2024-03-26T21:48:49.358 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.329954+0000", 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1500000000000005e-06 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.15804.0:91260 11.4 e459/433 11:218c6841:::benchmark_data_smithi055_186849_object91259:head v 459'5783, mlcod=459'5781)", 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.331961+0000", 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.0083268279999999997, 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0088252460000000001, 2024-03-26T21:48:49.359 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331961+0000", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331961+0000", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.360 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331962+0000", 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5799999999999999e-07 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331997+0000", 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5184000000000001e-05 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.332009+0000", 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2607e-05 2024-03-26T21:48:49.361 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.332016+0000", 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9190000000000002e-06 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.332218+0000", 2024-03-26T21:48:49.362 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00020157899999999999 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.332331+0000", 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113418 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.15804.0:91239 11.2 e459/433)", 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.338021+0000", 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.0022667550000000001, 2024-03-26T21:48:49.363 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028328950000000002, 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "queued for pg", 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338021+0000", 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338021+0000", 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.364 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338022+0000", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.5700000000000002e-07 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338026+0000", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7780000000000001e-06 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338041+0000", 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5519000000000002e-05 2024-03-26T21:48:49.365 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338050+0000", 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5660000000000007e-06 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.15804.0:91242 11.2 e459/433)", 2024-03-26T21:48:49.367 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.338071+0000", 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.0022164509999999999, 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028403500000000002, 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "queued for pg", 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338071+0000", 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.368 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338071+0000", 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338072+0000", 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.2399999999999999e-07 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338074+0000", 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.39e-06 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.369 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338079+0000", 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.045e-06 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338084+0000", 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5329999999999996e-06 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop_reply(client.15804.0:91239 11.2 e459/433)", 2024-03-26T21:48:49.370 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.338239+0000", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.002048724, 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002725562, 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "queued for pg", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338239+0000", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338239+0000", 2024-03-26T21:48:49.371 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338240+0000", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8500000000000001e-07 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338243+0000", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7479999999999999e-06 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338254+0000", 2024-03-26T21:48:49.372 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0725e-05 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.338260+0000", 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1809999999999998e-06 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.15804.0:91263 11.6 e459/433 11:6ecd0dd2:::benchmark_data_smithi055_186849_object91262:head v 459'5625, mlcod=459'5623)", 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.339513+0000", 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.00077521899999999995, 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0015107040000000001, 2024-03-26T21:48:49.373 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339513+0000", 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339513+0000", 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.374 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339513+0000", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7700000000000004e-07 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339535+0000", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1739e-05 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339551+0000", 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6314e-05 2024-03-26T21:48:49.375 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339559+0000", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3699999999999997e-06 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339715+0000", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000156107 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.339734+0000", 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9304e-05 2024-03-26T21:48:49.376 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:91239 11.2 11:420beca8:::benchmark_data_smithi055_186849_object91238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:48:49.310614+0000", 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.029673227999999999, 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.030485961999999998, 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:48:49.377 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 91239 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310614+0000", 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310614+0000", 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.378 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310616+0000", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.404e-06 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310637+0000", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.137e-05 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310649+0000", 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1945e-05 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.379 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310658+0000", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.251e-06 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.310814+0000", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015508300000000001 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.316867+0000", 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0060536710000000001 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.380 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.317062+0000", 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019455399999999999 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:48:49.331459+0000", 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.014396832 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 11 2024-03-26T21:48:49.381 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:48:49.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:49.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:49.534 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:49.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:48:49.597 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[40587]: pgmap v965: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 KiB/s rd, 78 MiB/s wr, 1.33k op/s 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[40587]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[29406]: pgmap v965: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 KiB/s rd, 78 MiB/s wr, 1.33k op/s 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:48:49.598 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:49 smithi055 ceph-mon[29406]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:48:49.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:49 smithi067 ceph-mon[33346]: pgmap v965: 148 pgs: 148 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 50 KiB/s rd, 78 MiB/s wr, 1.33k op/s 2024-03-26T21:48:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:49 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:48:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:49 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:48:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:49 smithi067 ceph-mon[33346]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T21:48:49.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.801 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:49.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:49.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:49.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 58 16 91828 91812 98.9044 61.4375 0.0127785 0.0101055 2024-03-26T21:48:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:49.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:50.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:50.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:50.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:50.326 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:50.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:50.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.605 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:50.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:50.706 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:50 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:50 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:50 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:50.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:50.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:48:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:48:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:50 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:48:50.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:50.879 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:50.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:50.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 59 16 92905 92889 98.369 67.3125 0.01043 0.0101609 2024-03-26T21:48:50.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:51.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:51.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:51.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:48:51.208 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:51.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.314 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:51.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:51.415 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:51.465 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:51.566 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:51.608 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:51 smithi055 ceph-mon[29406]: pgmap v967: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 KiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-03-26T21:48:51.609 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:51 smithi055 ceph-mon[40587]: pgmap v967: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 KiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-03-26T21:48:51.615 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:51.717 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:51 smithi067 ceph-mon[33346]: pgmap v967: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 KiB/s rd, 58 MiB/s wr, 1.03k op/s 2024-03-26T21:48:51.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:51.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:51.909 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:51.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:48:51.917329+0000 min lat: 0.00133566 max lat: 0.179135 avg lat: 0.010222 2024-03-26T21:48:51.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:48:51.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 60 16 93909 93893 97.7754 62.75 0.0209869 0.010222 2024-03-26T21:48:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:51.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:52.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:52.047 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:52.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:48:52.128 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:52.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:52.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:52.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:52.474 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:52.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:52.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:52.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:52.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:52.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:52.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:52.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 61 16 94984 94968 97.2739 67.1875 0.0133583 0.0102744 2024-03-26T21:48:52.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:53.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:53.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:52 smithi055 ceph-mon[29406]: pgmap v968: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 KiB/s rd, 54 MiB/s wr, 955 op/s 2024-03-26T21:48:53.052 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:52 smithi055 ceph-mon[29406]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:48:53.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:52 smithi055 ceph-mon[40587]: pgmap v968: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 KiB/s rd, 54 MiB/s wr, 955 op/s 2024-03-26T21:48:53.052 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:52 smithi055 ceph-mon[40587]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:48:53.116 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:53.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:52 smithi067 ceph-mon[33346]: pgmap v968: 148 pgs: 148 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 KiB/s rd, 54 MiB/s wr, 955 op/s 2024-03-26T21:48:53.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:52 smithi067 ceph-mon[33346]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T21:48:53.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:53.304 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:53.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:53.447 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:53.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:53.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:53.726 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:53.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:53.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 62 16 95959 95943 96.6879 60.9375 0.00956948 0.0103376 2024-03-26T21:48:53.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:53.982 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:53.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:54.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:54.126 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:54.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:54.261 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:54.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:54.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:54.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:54.624 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:54.662 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:54.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:54.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:54.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:54.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 63 16 97083 97067 96.2682 70.25 0.00442696 0.0103788 2024-03-26T21:48:54.928 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:54.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:55.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:55.076 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:55.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:55.216 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:55.220 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:54 smithi055 ceph-mon[40587]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 38 KiB/s rd, 89 MiB/s wr, 1.49k op/s 2024-03-26T21:48:55.220 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:54 smithi055 ceph-mon[29406]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 38 KiB/s rd, 89 MiB/s wr, 1.49k op/s 2024-03-26T21:48:55.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:54 smithi067 ceph-mon[33346]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 38 KiB/s rd, 89 MiB/s wr, 1.49k op/s 2024-03-26T21:48:55.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:55.339 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:55.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:55.453 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:48:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:55.496 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:48:55.529 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:48:55.541 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:48:55.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:55.599 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:48:55.640 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:48:55.644 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:55.684 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:48:55.725 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:48:55.747 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:55.769 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:48:55.776 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:48:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.796 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:55.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:55.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:55.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 64 16 98102 98086 95.7592 63.6875 0.0155588 0.010438 2024-03-26T21:48:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:55.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:56.035 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:56.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/324595472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:56.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/324595472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:56.072 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:56.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:56.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:56.213 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:56.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:56.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/324595472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:48:56.314 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:56.568 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:56.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:56.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:56.702 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:56.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:56.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:56.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:56.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:56.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 65 16 99130 99114 95.2744 64.25 0.0108495 0.010491 2024-03-26T21:48:56.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:57.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:57.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:57.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:56 smithi067 ceph-mon[33346]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 18 KiB/s rd, 65 MiB/s wr, 1.07k op/s 2024-03-26T21:48:57.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:57.289 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:57.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:56 smithi055 ceph-mon[29406]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 18 KiB/s rd, 65 MiB/s wr, 1.07k op/s 2024-03-26T21:48:57.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:48:57.114+0000 7fb1ff446700 -1 mon.a@0(leader).osd e460 definitely_dead 0 2024-03-26T21:48:57.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:56 smithi055 ceph-mon[40587]: pgmap v971: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 18 KiB/s rd, 65 MiB/s wr, 1.07k op/s 2024-03-26T21:48:57.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:57.428 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:57.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:57.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.696 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:57.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:57.797 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:57.832 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:57.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 66 15 100185 100170 94.8309 66 0.00858916 0.0105397 2024-03-26T21:48:57.934 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:57.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:48:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:48:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:57.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:58.080 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.105 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:58.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:58.206 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:58.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3750466864' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:57 smithi067 ceph-mon[33346]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-26T21:48:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3750466864' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[40587]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-26T21:48:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3750466864' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-26T21:48:58.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:57 smithi055 ceph-mon[29406]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-26T21:48:58.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:58.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:58.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:48:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:58.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:48:58.587 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:58.614 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:58.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:48:58.744 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:48:58.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:58.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:58.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:58.909 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 67 16 100251 100235 93.4764 4.0625 0.00326908 0.0105437 2024-03-26T21:48:58.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:58.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:48:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:59.108 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:59.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:59.181 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[40587]: pgmap v972: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.9 KiB/s rd, 52 MiB/s wr, 838 op/s 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[40587]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[40587]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[29406]: pgmap v972: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.9 KiB/s rd, 52 MiB/s wr, 838 op/s 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[29406]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T21:48:59.182 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-mon[29406]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T21:48:59.183 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:48:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:48:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:48:59.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:48:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:58 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:48:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:58 smithi067 ceph-mon[33346]: pgmap v972: 148 pgs: 148 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.9 KiB/s rd, 52 MiB/s wr, 838 op/s 2024-03-26T21:48:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-26T21:48:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:58 smithi067 ceph-mon[33346]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T21:48:59.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:58 smithi067 ceph-mon[33346]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T21:48:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:48:59.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:48:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:48:59.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:48:59.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:48:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:48:59.627 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:48:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.652 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:59.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:48:59.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:48:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.777 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:59.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:48:59.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:48:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:48:59.904 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:48:59.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:48:59.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 68 16 100251 100235 92.102 0 - 0.0105437 2024-03-26T21:49:00.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:00.031 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:00.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:00.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:00.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:59 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:00.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:59 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:00.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:59 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:00.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:48:59 smithi067 ceph-mon[33346]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T21:49:00.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:00.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:00.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:48:59 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:48:59 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2035370726' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:00.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T21:49:00.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.443 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:00.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:00.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.587 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:00.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:00.688 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:00.721 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:00.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:00.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:00.894 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:49:00.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 69 15 100814 100799 91.2781 17.625 0.0112621 0.0109499 2024-03-26T21:49:00.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:01.003 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:01.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:01.142 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:01.209 INFO:teuthology.orchestra.run.smithi055.stderr:marked out osd.2. 2024-03-26T21:49:01.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: pgmap v975: 148 pgs: 1 remapped+peering, 1 unknown, 21 peering, 125 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:49:01.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 460'6341 2024-03-26T21:49:01.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 460'6172 2024-03-26T21:49:01.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 460'6206 2024-03-26T21:49:01.209 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 460'6339 2024-03-26T21:49:01.210 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: pgmap v975: 148 pgs: 1 remapped+peering, 1 unknown, 21 peering, 125 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 460'6341 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 460'6172 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 460'6206 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 460'6339 2024-03-26T21:49:01.211 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:00 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: pgmap v975: 148 pgs: 1 remapped+peering, 1 unknown, 21 peering, 125 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: 11.c starting backfill to osd.7 from (0'0,0'0] MAX to 460'6341 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 460'6172 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 460'6206 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 460'6339 2024-03-26T21:49:01.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:00 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:01.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:01.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:01.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.423 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:01.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:01.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:01.697 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:01.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:01.832 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:01.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 70 16 101747 101731 90.8063 58.25 0.0172222 0.0110054 2024-03-26T21:49:01.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:01.974 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:01.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:02.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.115 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:02.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:02.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:02.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:02.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:02.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:02.544 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:02.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:02.688 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:02.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:02.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:02.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 71 16 102700 102684 90.3663 59.5625 0.0115505 0.0110612 2024-03-26T21:49:02.942 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:02.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:02.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:02.993 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[40587]: pgmap v977: 148 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 21 peering, 124 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 55 MiB/s wr, 881 op/s; 6248/301086 objects misplaced (2.075%); 139 KiB/s, 4 objects/s recovering 2024-03-26T21:49:02.994 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[29406]: pgmap v977: 148 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 21 peering, 124 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 55 MiB/s wr, 881 op/s; 6248/301086 objects misplaced (2.075%); 139 KiB/s, 4 objects/s recovering 2024-03-26T21:49:02.994 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:02.994 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:03.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:03.122 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:03.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:03.261 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:03.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:03.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:03.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:03.419 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:02 smithi067 ceph-mon[33346]: pgmap v977: 148 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 unknown, 21 peering, 124 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 55 MiB/s wr, 881 op/s; 6248/301086 objects misplaced (2.075%); 139 KiB/s, 4 objects/s recovering 2024-03-26T21:49:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:02 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:03.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:02 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:03.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:03.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:03.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.826 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:03.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:03.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 72 16 103615 103599 89.9055 57.1875 0.0252322 0.0111158 2024-03-26T21:49:03.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:03.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:03.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:04.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:04.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:04.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:04.247 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:04.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:04 smithi055 ceph-mon[29406]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:49:04.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:04 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:49:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:04 smithi055 ceph-mon[40587]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:49:04.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:04 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:49:04.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:04.386 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:04.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:04 smithi067 ceph-mon[33346]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T21:49:04.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:04 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T21:49:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:04.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:04.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:04.802 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:49:04.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:04.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 73 15 104528 104513 89.4564 57.125 0.0182372 0.0111719 2024-03-26T21:49:04.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:04.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:04.980 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:49:04.980 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:49:05.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:05.087 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:05.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:05.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:05 smithi055 ceph-mon[29406]: pgmap v978: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 remapped+peering, 142 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 913 op/s; 31028/307542 objects misplaced (10.089%); 10 MiB/s, 173 objects/s recovering 2024-03-26T21:49:05.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:05 smithi055 ceph-mon[40587]: pgmap v978: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 remapped+peering, 142 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 913 op/s; 31028/307542 objects misplaced (10.089%); 10 MiB/s, 173 objects/s recovering 2024-03-26T21:49:05.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:05.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:05.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:05 smithi067 ceph-mon[33346]: pgmap v978: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 remapped+peering, 142 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 57 MiB/s wr, 913 op/s; 31028/307542 objects misplaced (10.089%); 10 MiB/s, 173 objects/s recovering 2024-03-26T21:49:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.509 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:05.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:05.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:05.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:05.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:05.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:05.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:05.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 74 16 105412 105396 88.9933 55.1875 0.0218399 0.0112308 2024-03-26T21:49:05.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:05.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:06.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.065 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:06.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:06.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:06.180 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:06 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:06.180 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:06 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:06.180 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:06 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:06.180 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:06 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:06.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:06.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:49:06.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:06.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:06.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:06.488 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:06.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:06 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:06.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:06 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:06.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.637 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:06.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:06.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.773 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:06.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:06.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:06.909 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:06.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:06.910 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 75 16 106255 106239 88.5087 52.6875 0.0154753 0.0112906 2024-03-26T21:49:06.930 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:49:06.957 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:06.987 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:07.005 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:49:07.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:49:07.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:49:07.010 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T21:49:07.010 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-03-26T21:49:07.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 6 0 2024-03-26T21:49:07.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:07.047 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:49:07.094 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:49:07.106 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:07.138 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:49:07.184 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:49:07.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:07.229 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:49:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:07.278 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:49:07.287 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:49:07.288 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:49:07.288 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[40587]: pgmap v979: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 734 op/s; 36856/308238 objects misplaced (11.957%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[40587]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2339761935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[29406]: pgmap v979: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 734 op/s; 36856/308238 objects misplaced (11.957%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[29406]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T21:49:07.320 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2339761935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:07.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:07.398 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:07 smithi067 ceph-mon[33346]: pgmap v979: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 734 op/s; 36856/308238 objects misplaced (11.957%); 13 MiB/s, 207 objects/s recovering 2024-03-26T21:49:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:07 smithi067 ceph-mon[33346]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T21:49:07.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2339761935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:07.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:07.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:07.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:07.910 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:07.911 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 76 16 107033 107017 87.984 48.625 0.0347926 0.0113597 2024-03-26T21:49:07.952 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:07.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:07.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:08.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:08.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:08.191 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:08.233 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:08.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:08.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:08.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:08.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:08.666 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:08.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:08.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:08.911 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 77 16 107921 107905 87.5616 55.5 0.00589082 0.0114149 2024-03-26T21:49:08.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:08.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:09.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:09.080 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:09.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.213 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:09.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:09.314 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:09.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:09 smithi055 ceph-mon[40587]: pgmap v981: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 36757/312375 objects misplaced (11.767%); 13 MiB/s, 213 objects/s recovering 2024-03-26T21:49:09.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:09.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:09 smithi055 ceph-mon[29406]: pgmap v981: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 36757/312375 objects misplaced (11.767%); 13 MiB/s, 213 objects/s recovering 2024-03-26T21:49:09.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:09.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:09.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:09.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:09 smithi067 ceph-mon[33346]: pgmap v981: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 503 op/s; 36757/312375 objects misplaced (11.767%); 13 MiB/s, 213 objects/s recovering 2024-03-26T21:49:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:09.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:09.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.653 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:09.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:09.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:09.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:09.880 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:09.905 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:09.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:09.911 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 78 16 108651 108635 87.024 45.625 0.00399754 0.0114805 2024-03-26T21:49:10.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:10.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:10.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:10.189 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:10.291 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:10.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:10.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:10.468 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:10.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:10.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:10.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:10.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:10.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:10.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:10.911 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 79 16 109563 109547 86.644 57 0.00499504 0.011535 2024-03-26T21:49:10.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:11.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:11.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:11.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:11.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:11.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:11.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:11.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:11.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:11 smithi067 ceph-mon[33346]: pgmap v982: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 721 op/s; 35520/321441 objects misplaced (11.050%); 25 MiB/s, 400 objects/s recovering 2024-03-26T21:49:11.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:11.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:11 smithi055 ceph-mon[29406]: pgmap v982: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 721 op/s; 35520/321441 objects misplaced (11.050%); 25 MiB/s, 400 objects/s recovering 2024-03-26T21:49:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:11 smithi055 ceph-mon[40587]: pgmap v982: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 721 op/s; 35520/321441 objects misplaced (11.050%); 25 MiB/s, 400 objects/s recovering 2024-03-26T21:49:11.576 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:11.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:11.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:11.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.855 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:11.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:11.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:2024-03-26T21:49:11.920075+0000 min lat: 0.00133566 max lat: 2.44144 avg lat: 0.0115998 2024-03-26T21:49:11.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T21:49:11.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 80 16 110348 110332 86.1742 49.0625 0.0111408 0.0115998 2024-03-26T21:49:11.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:11.990 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:11.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:12.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.127 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:12.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:12.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:12.270 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:12.372 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:12.407 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:12.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.549 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:12.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:12.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:12.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:12.816 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:12.912 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 81 16 111214 111198 85.7786 54.125 0.0104093 0.0116512 2024-03-26T21:49:12.917 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:12.957 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:12.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:13.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:13.113 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:13.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:13.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:13.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:13 smithi055 ceph-mon[40587]: pgmap v983: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 728 op/s; 35429/322917 objects misplaced (10.972%); 25 MiB/s, 398 objects/s recovering 2024-03-26T21:49:13.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:13 smithi055 ceph-mon[29406]: pgmap v983: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 728 op/s; 35429/322917 objects misplaced (10.972%); 25 MiB/s, 398 objects/s recovering 2024-03-26T21:49:13.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:13.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:13.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:13 smithi067 ceph-mon[33346]: pgmap v983: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 45 MiB/s wr, 728 op/s; 35429/322917 objects misplaced (10.972%); 25 MiB/s, 398 objects/s recovering 2024-03-26T21:49:13.495 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:13.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:13.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:13.673 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:13.775 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:13.818 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:13.912 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 82 15 112067 112052 85.3834 53.375 0.0477139 0.0117066 2024-03-26T21:49:13.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:13.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:14.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:14.089 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:14.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:49:14.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:14.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:14.323 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.368 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:14.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:14.401 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:49:14.401 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:49:14.401 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:49:14.401 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:2867 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94208173336832 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:05.658367+0000", 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 488.73649056400001, 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0722914889999999, 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 2867 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.402 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658367+0000", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658367+0000", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658368+0000", 2024-03-26T21:49:14.403 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2899999999999999e-06 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658375+0000", 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2190000000000001e-06 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658390+0000", 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5160999999999999e-05 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.404 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658400+0000", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6600000000000007e-06 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658469+0000", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9609000000000004e-05 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.658476+0000", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9779999999999999e-06 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.405 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.670325+0000", 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011848757999999999 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:05.670334+0000", 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7879999999999995e-06 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.674349+0000", 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0040157220000001 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.406 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.674361+0000", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2091e-05 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.722686+0000", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.048324931 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for degraded object", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.723057+0000", 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000370276 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.407 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.726551+0000", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034944260000000001 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.726650+0000", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8597000000000005e-05 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.730659+0000", 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040089829999999998 2024-03-26T21:49:14.408 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:2869 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94208173336832 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:06.689371+0000", 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 487.70548710000003, 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0418744659999999, 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:49:14.409 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 2869 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689371+0000", 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689371+0000", 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.410 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689372+0000", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7969999999999999e-06 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689377+0000", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2830000000000003e-06 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689386+0000", 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4879999999999996e-06 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.411 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.689393+0000", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9840000000000004e-06 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.690067+0000", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00067399199999999999 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:06.690077+0000", 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7249999999999992e-06 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.412 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.723089+0000", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0330121189999999 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for degraded object", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.723131+0000", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2425999999999997e-05 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.727117+0000", 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0039859479999999996 2024-03-26T21:49:14.413 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.727197+0000", 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9140000000000005e-05 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:07.731245+0000", 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040485640000000002 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.414 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:2903 5.d 5:b4812045:::notify.4:head [watch ping cookie 94208173346048 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.044329+0000", 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 469.35052876200001, 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.56792915499999996, 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 2903 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.415 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044329+0000", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044329+0000", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044330+0000", 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4020000000000001e-06 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.416 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044343+0000", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2850999999999999e-05 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044358+0000", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4599e-05 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044366+0000", 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6869999999999994e-06 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.417 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044632+0000", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00026681799999999999 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044644+0000", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.129e-05 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.612002+0000", 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.56735856600000001 2024-03-26T21:49:14.418 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.612187+0000", 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018430599999999999 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.612258+0000", 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1636000000000003e-05 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.419 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:2911 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94208173346048 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:25.044837+0000", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 469.35002106299999, 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.570088445, 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 2911 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.420 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044837+0000", 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044837+0000", 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044837+0000", 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1699999999999997e-07 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.421 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044841+0000", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6839999999999998e-06 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044850+0000", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7530000000000007e-06 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044857+0000", 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3409999999999997e-06 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.422 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044905+0000", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7466000000000003e-05 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.044914+0000", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.969e-06 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.612321+0000", 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.567407208 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.423 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.612395+0000", 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3983e-05 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:25.614925+0000", 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0025302240000000002 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:52112 10.3 10:c3eed624:::benchmark_data_smithi055_163313_object52111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:49:14.424 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:44:04.512569+0000", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 309.88228849299998, 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.85595948300000002, 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 52112 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512569+0000", 2024-03-26T21:49:14.425 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512569+0000", 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512571+0000", 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.666e-06 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.426 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512595+0000", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4468e-05 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512607+0000", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1824999999999999e-05 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512616+0000", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6670000000000002e-06 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.427 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512677+0000", 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1266000000000006e-05 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512686+0000", 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2e-06 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.352368+0000", 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.83968198500000002 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.428 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.352479+0000", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011033 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 4,5", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.352612+0000", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013383199999999999 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.354122+0000", 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0015097579999999999 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.429 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.362921+0000", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087988259999999992 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.368435+0000", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0055139100000000003 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.368487+0000", 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1549999999999999e-05 2024-03-26T21:49:14.430 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.368529+0000", 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2200000000000003e-05 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111427 10.5 10:a5f48779:::benchmark_data_smithi055_163313_object7378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.429424+0000", 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.965433249, 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0341454809999999, 2024-03-26T21:49:14.431 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111427 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429424+0000", 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.432 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429424+0000", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429425+0000", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9100000000000004e-07 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429428+0000", 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.41e-06 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.433 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429435+0000", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9589999999999998e-06 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429442+0000", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1929999999999998e-06 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429503+0000", 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.1662999999999996e-05 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.434 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.429514+0000", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0764999999999999e-05 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.449777+0000", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.020262612 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.449894+0000", 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000117862 2024-03-26T21:49:14.435 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.449996+0000", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000101561 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461060+0000", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011063931000000001 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461119+0000", 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8659999999999997e-05 2024-03-26T21:49:14.436 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463348+0000", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0022296769999999998 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463517+0000", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00016865599999999999 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463570+0000", 2024-03-26T21:49:14.437 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3041000000000003e-05 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111433 10.5 10:a2ff991b:::benchmark_data_smithi055_163313_object7384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.441989+0000", 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.95286855399999, 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.028500441, 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.438 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111433 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.441989+0000", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.441989+0000", 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.439 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.441991+0000", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.505e-06 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.441997+0000", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6239999999999996e-06 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.442095+0000", 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7653999999999997e-05 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.440 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.442104+0000", 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0510000000000002e-06 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.441 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.442652+0000", 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00054789400000000005 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.442663+0000", 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1419e-05 2024-03-26T21:49:14.442 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.452675+0000", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0100113770000001 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.452796+0000", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121272 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.452909+0000", 2024-03-26T21:49:14.443 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011296199999999999 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461102+0000", 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081934650000000005 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463610+0000", 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0025080810000000001 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.444 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470402+0000", 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0067919260000000002 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470448+0000", 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5742999999999999e-05 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470490+0000", 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1467999999999997e-05 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.445 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111448 10.5 10:a4a4a5ad:::benchmark_data_smithi055_163313_object7399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.455487+0000", 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.93937072599999, 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0152809, 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111448 2024-03-26T21:49:14.446 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455487+0000", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455487+0000", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455489+0000", 2024-03-26T21:49:14.447 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9350000000000001e-06 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455494+0000", 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3890000000000004e-06 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455507+0000", 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2638e-05 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.448 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455516+0000", 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0420000000000008e-06 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455613+0000", 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7137000000000002e-05 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455625+0000", 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1868999999999999e-05 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.449 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.454998+0000", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99937258299999998 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.455126+0000", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012880699999999999 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.455252+0000", 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000125919 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.450 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463649+0000", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0083966100000000005 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470498+0000", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0068493269999999997 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470698+0000", 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00019984399999999999 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.451 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470736+0000", 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7978000000000001e-05 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470768+0000", 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.1822000000000001e-05 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111449 10.5 10:a64c14ae:::benchmark_data_smithi055_163313_object7400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.452 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.455543+0000", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.93931488600001, 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0153869470000001, 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111449 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455543+0000", 2024-03-26T21:49:14.453 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455543+0000", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455543+0000", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.63e-07 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455546+0000", 2024-03-26T21:49:14.454 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9500000000000001e-06 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455553+0000", 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4370000000000004e-06 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455558+0000", 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7910000000000001e-06 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.455 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455650+0000", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1871000000000002e-05 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.455658+0000", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3080000000000002e-06 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.457175+0000", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0015173449999999 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.456 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.457358+0000", 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018260999999999999 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.457476+0000", 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011796800000000001 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463684+0000", 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0062083470000000003 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.457 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470505+0000", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0068205840000000002 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470816+0000", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00031122700000000002 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470887+0000", 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1243999999999998e-05 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.458 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470930+0000", 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2802000000000001e-05 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111519 10.5 10:a3bb0683:::benchmark_data_smithi055_163313_object7470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.533221+0000", 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.861636664, 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.93787931899999999, 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.459 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111519 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533221+0000", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533221+0000", 2024-03-26T21:49:14.460 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533223+0000", 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8109999999999999e-06 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533227+0000", 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1359999999999999e-06 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.461 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533237+0000", 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0071000000000001e-05 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533244+0000", 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2200000000000003e-06 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533337+0000", 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3140999999999995e-05 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.462 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.533348+0000", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0294e-05 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.459361+0000", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92601340899999995 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.459475+0000", 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011414200000000001 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.463 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.459577+0000", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000101937 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470546+0000", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010968492999999999 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470598+0000", 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2364999999999998e-05 2024-03-26T21:49:14.464 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470982+0000", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00038342699999999999 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471028+0000", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6190000000000003e-05 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471100+0000", 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2682999999999995e-05 2024-03-26T21:49:14.465 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111524 10.5 10:a4873c79:::benchmark_data_smithi055_163313_object7475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.542265+0000", 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.85259287299999, 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92900779499999997, 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.466 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111524 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542265+0000", 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542265+0000", 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.467 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542266+0000", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2589999999999999e-06 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542271+0000", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.994e-06 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542281+0000", 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.4870000000000008e-06 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.468 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542288+0000", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9789999999999996e-06 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542348+0000", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0550999999999997e-05 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.542357+0000", 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0270000000000001e-06 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.469 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461332+0000", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91897442200000001 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461484+0000", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000152857 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.461594+0000", 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000109969 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.470 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470587+0000", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0089921949999999997 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470604+0000", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7108e-05 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471196+0000", 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00059268999999999999 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.471 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471237+0000", 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1066999999999998e-05 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471273+0000", 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.519e-05 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111644 10.5 10:a2941a33:::benchmark_data_smithi055_163313_object7595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.472 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.704706+0000", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.69015133900001, 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.77495886999999997, 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111644 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704706+0000", 2024-03-26T21:49:14.473 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704706+0000", 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704708+0000", 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5659999999999999e-06 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.474 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704714+0000", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7620000000000001e-06 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704835+0000", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121606 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704851+0000", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5341000000000001e-05 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.475 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704930+0000", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9635000000000006e-05 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.704941+0000", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0210000000000001e-05 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463711+0000", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.75877070300000005 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.476 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463826+0000", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000115117 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.463936+0000", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010921000000000001 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470609+0000", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.006673376 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.470651+0000", 2024-03-26T21:49:14.477 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1980000000000001e-05 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479549+0000", 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0088980050000000005 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479623+0000", 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4522000000000002e-05 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.478 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479665+0000", 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1836999999999999e-05 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111665 10.5 10:a4b61831:::benchmark_data_smithi055_163313_object7616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.725290+0000", 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.66956809600001, 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.75448845899999994, 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.479 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111665 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725290+0000", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725290+0000", 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.480 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725291+0000", 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3209999999999999e-06 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725297+0000", 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0000000000000002e-06 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.481 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725313+0000", 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.645e-05 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725323+0000", 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2890000000000003e-06 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725662+0000", 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00033943299999999999 2024-03-26T21:49:14.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.725673+0000", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1042e-05 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.465911+0000", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.740237743 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.466028+0000", 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011666 2024-03-26T21:49:14.484 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.466181+0000", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00015387500000000001 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471312+0000", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0051303290000000003 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479504+0000", 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081918019999999998 2024-03-26T21:49:14.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479709+0000", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00020515999999999999 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479744+0000", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4749000000000002e-05 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479778+0000", 2024-03-26T21:49:14.486 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4606e-05 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:111701 10.5 10:a7477d4d:::benchmark_data_smithi055_163313_object7652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:20.762328+0000", 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 233.632529957, 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.73115565699999996, 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.487 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 111701 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762328+0000", 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762328+0000", 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.488 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762329+0000", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4700000000000001e-07 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762333+0000", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.031e-06 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762341+0000", 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2269999999999993e-06 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.489 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762346+0000", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5999999999999997e-06 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762411+0000", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.4128999999999995e-05 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:20.762419+0000", 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2099999999999993e-06 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.490 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471352+0000", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.70893298299999996 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471479+0000", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012757900000000001 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,3", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.471580+0000", 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010064300000000001 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.479508+0000", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0079284149999999994 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.483995+0000", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0044863350000000001 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.493342+0000", 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0093469720000000003 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.492 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.493431+0000", 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8867000000000001e-05 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.493483+0000", 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2918999999999999e-05 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143343 10.18 10:1bcfd877:::benchmark_data_smithi055_163313_object39294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.493 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:11.906652+0000", 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.488206101, 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96630665500000001, 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143343 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.494 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906652+0000", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906652+0000", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906653+0000", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0759999999999999e-06 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.495 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906657+0000", 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0570000000000003e-06 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906666+0000", 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3070000000000008e-06 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906673+0000", 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.277e-06 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.496 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906732+0000", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8498000000000001e-05 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:11.906742+0000", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7200000000000001e-06 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.867403+0000", 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96066161000000005 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.497 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.867558+0000", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000154432 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.867702+0000", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000144406 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.870959+0000", 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032566869999999999 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.498 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.871016+0000", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6969000000000003e-05 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.872864+0000", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001848768 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.872913+0000", 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8430000000000002e-05 2024-03-26T21:49:14.499 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.872958+0000", 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5417999999999998e-05 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143370 10.18 10:1e83733f:::benchmark_data_smithi055_163313_object39321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:12.040578+0000", 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.35428002699999, 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.83585746000000005, 2024-03-26T21:49:14.500 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143370 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040578+0000", 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.501 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040578+0000", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040579+0000", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1769999999999999e-06 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040584+0000", 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8840000000000002e-06 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.502 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040595+0000", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1258e-05 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040602+0000", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7510000000000002e-06 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040658+0000", 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5967999999999997e-05 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.503 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.040667+0000", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3980000000000005e-06 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.871054+0000", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.83038730699999996 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.871143+0000", 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8928e-05 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.504 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.871231+0000", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7779000000000001e-05 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876297+0000", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.005065941 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876343+0000", 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5980999999999999e-05 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.505 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876351+0000", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3049999999999993e-06 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876379+0000", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7305999999999999e-05 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876435+0000", 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6477000000000002e-05 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.506 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143412 10.18 10:1825125b:::benchmark_data_smithi055_163313_object39363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:12.136924+0000", 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.257933533, 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.73969525800000002, 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143412 2024-03-26T21:49:14.507 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136924+0000", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136924+0000", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136925+0000", 2024-03-26T21:49:14.508 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.850000000000001e-07 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136930+0000", 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9590000000000003e-06 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136942+0000", 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1683e-05 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.509 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.136950+0000", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5639999999999997e-06 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.137020+0000", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9109000000000005e-05 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.137034+0000", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4404e-05 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.510 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.872994+0000", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.73596026699999995 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.873111+0000", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000116483 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.873210+0000", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9011999999999993e-05 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.511 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876446+0000", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0032365760000000001 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876486+0000", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9379999999999999e-05 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876548+0000", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2067000000000001e-05 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.512 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876581+0000", 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3633000000000001e-05 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876619+0000", 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8136000000000002e-05 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143417 10.18 10:1cb0da3a:::benchmark_data_smithi055_163313_object39368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:12.149987+0000", 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.244871227, 2024-03-26T21:49:14.513 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.72687409199999997, 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143417 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.149987+0000", 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.514 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.149987+0000", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.149989+0000", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0269999999999998e-06 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.149995+0000", 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7750000000000004e-06 2024-03-26T21:49:14.515 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.150020+0000", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4201000000000001e-05 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.150034+0000", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4212999999999999e-05 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.150123+0000", 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.8801999999999994e-05 2024-03-26T21:49:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.150140+0000", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7206000000000001e-05 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.874684+0000", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.72454386800000004 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.874779+0000", 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5136000000000005e-05 2024-03-26T21:49:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.874869+0000", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0111999999999994e-05 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876499+0000", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001629889 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876675+0000", 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000176293 2024-03-26T21:49:14.518 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876773+0000", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7663000000000003e-05 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876822+0000", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9404000000000003e-05 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876861+0000", 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8503e-05 2024-03-26T21:49:14.519 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143434 10.18 10:1a8c2190:::benchmark_data_smithi055_163313_object39385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:12.219987+0000", 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.1748709, 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.66380256000000004, 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.520 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143434 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.219987+0000", 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.219987+0000", 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.521 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.522 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.522 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.522 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.219988+0000", 2024-03-26T21:49:14.522 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1699999999999997e-07 2024-03-26T21:49:14.522 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.219992+0000", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2309999999999998e-06 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.220001+0000", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5929999999999995e-06 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.220008+0000", 2024-03-26T21:49:14.523 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5119999999999997e-06 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.220055+0000", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6712000000000003e-05 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.220063+0000", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0050000000000004e-06 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.876922+0000", 2024-03-26T21:49:14.524 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.65685923599999996 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.877046+0000", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000123724 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.877170+0000", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000124222 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883617+0000", 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0064473100000000004 2024-03-26T21:49:14.525 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883673+0000", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.605e-05 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883691+0000", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7450999999999999e-05 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883741+0000", 2024-03-26T21:49:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9796999999999998e-05 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883789+0000", 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8900000000000003e-05 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:143445 10.18 10:1f9b62e4:::benchmark_data_smithi055_163313_object39396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:12.302342+0000", 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 182.09251615700001, 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.58163810199999999, 2024-03-26T21:49:14.527 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 143445 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302342+0000", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:14.528 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302342+0000", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302343+0000", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.35e-06 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302348+0000", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3299999999999998e-06 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:14.529 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302361+0000", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2748999999999999e-05 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302369+0000", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7980000000000006e-06 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302428+0000", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8730000000000002e-05 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:14.530 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.302438+0000", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.04e-05 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.879026+0000", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.57658784500000004 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.879138+0000", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011184 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,3", 2024-03-26T21:49:14.531 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.879265+0000", 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012775899999999999 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883797+0000", 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004531836 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883846+0000", 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8613999999999999e-05 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.532 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883915+0000", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8764999999999996e-05 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883953+0000", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.8244999999999997e-05 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:12.883980+0000", 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.6840999999999999e-05 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.533 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:14.534 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:14.534 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:49:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:14.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:14.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:14.823 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:14.912 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 83 16 112798 112782 84.9044 45.625 0.0268118 0.0117723 2024-03-26T21:49:14.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:14.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:14.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:15.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.102 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:15.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:15.177 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:49:15.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:15.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[40587]: pgmap v984: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s; 34604/333105 objects misplaced (10.388%); 28 MiB/s, 445 objects/s recovering 2024-03-26T21:49:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3795289907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[29406]: pgmap v984: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s; 34604/333105 objects misplaced (10.388%); 28 MiB/s, 445 objects/s recovering 2024-03-26T21:49:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3795289907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:15 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:15.384 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:15.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:15 smithi067 ceph-mon[33346]: pgmap v984: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s; 34604/333105 objects misplaced (10.388%); 28 MiB/s, 445 objects/s recovering 2024-03-26T21:49:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:15 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3795289907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:15 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:49:15.525 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:15.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:49:15.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:15.665 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:15.767 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.814 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:15.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:15.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 84 16 113661 113645 84.535 53.9375 0.0139666 0.011823 2024-03-26T21:49:15.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:15.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:16.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:16.139 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:16.139 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:49:16.139 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[29406]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T21:49:16.185 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:16.222 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:16.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:16.354 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:16.358 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:16.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:16 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:16 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:49:16.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:16 smithi067 ceph-mon[33346]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T21:49:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:16.519 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:49:16.519 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:49:16.519 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:16 smithi055 ceph-mon[40587]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T21:49:16.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:16.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:16.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:16.776 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:16.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:16.904 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:16.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:16.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 85 16 114508 114492 84.1633 52.9375 0.0855564 0.0118695 2024-03-26T21:49:17.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:17.093 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:17.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:17.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:17.310 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:17.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:17.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:17 smithi067 ceph-mon[33346]: pgmap v986: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s; 33965/333945 objects misplaced (10.171%); 31 MiB/s, 488 objects/s recovering 2024-03-26T21:49:17.551 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:17.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:17 smithi055 ceph-mon[40587]: pgmap v986: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s; 33965/333945 objects misplaced (10.171%); 31 MiB/s, 488 objects/s recovering 2024-03-26T21:49:17.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:17 smithi055 ceph-mon[29406]: pgmap v986: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s; 33965/333945 objects misplaced (10.171%); 31 MiB/s, 488 objects/s recovering 2024-03-26T21:49:17.591 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:17.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:17.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:17.894 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:17.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:17.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 86 15 115332 115317 83.7842 51.5625 0.0134507 0.0119303 2024-03-26T21:49:17.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:17.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:18.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:18.177 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:18.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:18.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:49:18.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.458 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:18.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:18.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:18.590 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:18.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.729 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:18.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:18.830 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:18.858 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:18.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:18.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 87 16 116105 116089 83.3758 48.25 0.0827139 0.0119798 2024-03-26T21:49:18.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:19.007 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:19.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[40587]: pgmap v987: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 846 op/s; 33867/337749 objects misplaced (10.027%); 30 MiB/s, 472 objects/s recovering 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[29406]: pgmap v987: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 846 op/s; 33867/337749 objects misplaced (10.027%); 30 MiB/s, 472 objects/s recovering 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:19.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:19.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:19.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:18 smithi067 ceph-mon[33346]: pgmap v987: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 53 MiB/s wr, 846 op/s; 33867/337749 objects misplaced (10.027%); 30 MiB/s, 472 objects/s recovering 2024-03-26T21:49:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:19.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.268 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:19.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.408 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:19.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:19.554 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:19.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.686 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:19.787 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:19.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 88 16 116868 116852 82.9703 47.6875 0.0128006 0.012042 2024-03-26T21:49:19.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:19.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:20.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:20.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:20.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:20.258 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:20.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:20.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:20.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.538 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:20.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:20.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:20.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:20.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:20.818 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:20.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout: 89 16 117718 117702 82.635 53.125 0.0130763 0.0120965 2024-03-26T21:49:20.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:20.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:21.059 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[40587]: pgmap v988: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s; 32568/346335 objects misplaced (9.404%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:49:21.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:49:21.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[29406]: pgmap v988: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s; 32568/346335 objects misplaced (9.404%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:49:21.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:20 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:49:21.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:21.197 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.237 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:20 smithi067 ceph-mon[33346]: pgmap v988: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 830 op/s; 32568/346335 objects misplaced (9.404%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:20 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:49:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:20 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:49:21.264 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:21.338 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.384 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:21.485 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:21.530 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:21.631 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:21.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:21.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:21.825 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:21.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total time run: 90.0184 2024-03-26T21:49:21.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Total writes made: 118619 2024-03-26T21:49:21.913 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Write size: 65536 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Object size: 65536 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Bandwidth (MB/sec): 82.3575 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Bandwidth: 37.94 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max bandwidth (MB/sec): 216.688 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min bandwidth (MB/sec): 0 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average IOPS: 1317 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev IOPS: 607.041 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max IOPS: 3467 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min IOPS: 0 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Average Latency(s): 0.0121368 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Stddev Latency(s): 0.027918 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Max latency(s): 2.44144 2024-03-26T21:49:21.914 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Min latency(s): 0.00133566 2024-03-26T21:49:21.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:21.952 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T21:49:21.958 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:21.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:22.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:22.094 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:22.196 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.228 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:22.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:22.362 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:22.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:22.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:22.606 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:22.643 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:22.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:22.780 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:22.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:22.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:23.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:23.065 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:22 smithi055 ceph-mon[29406]: pgmap v989: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s; 32477/347772 objects misplaced (9.339%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:23.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:22 smithi055 ceph-mon[40587]: pgmap v989: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s; 32477/347772 objects misplaced (9.339%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:23.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.191 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:23.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:22 smithi067 ceph-mon[33346]: pgmap v989: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s; 32477/347772 objects misplaced (9.339%); 29 MiB/s, 460 objects/s recovering 2024-03-26T21:49:23.292 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.331 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:23.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:23.432 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:23.472 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:23.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.608 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:23.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:23.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:23.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:23.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:23.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:23.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:24.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.160 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:24.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:24.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:24.306 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:24.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:24.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:24.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:24.591 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:24.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:24.733 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:24.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:24.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:24.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:24.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:25.011 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:25.114 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:25.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:24 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:24 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:24 smithi067 ceph-mon[33346]: pgmap v990: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 47 MiB/s wr, 833 op/s; 31391/353391 objects misplaced (8.883%); 28 MiB/s, 450 objects/s recovering 2024-03-26T21:49:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:24 smithi067 ceph-mon[33346]: 11.7 scrub starts 2024-03-26T21:49:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:24 smithi067 ceph-mon[33346]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T21:49:25.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:25.294 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:25.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[40587]: pgmap v990: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 47 MiB/s wr, 833 op/s; 31391/353391 objects misplaced (8.883%); 28 MiB/s, 450 objects/s recovering 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[40587]: 11.7 scrub starts 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[40587]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[29406]: pgmap v990: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 47 MiB/s wr, 833 op/s; 31391/353391 objects misplaced (8.883%); 28 MiB/s, 450 objects/s recovering 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[29406]: 11.7 scrub starts 2024-03-26T21:49:25.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:24 smithi055 ceph-mon[29406]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T21:49:25.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.440 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:25.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:25.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:25.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:25.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:25.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:25.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:25.866 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:25.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:25.968 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:26.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:26.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:26.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:26.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:26.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:26.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:26.447 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:26.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:26.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:26.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:26.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:26.859 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:26.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:26.961 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:27.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:26 smithi055 ceph-mon[40587]: pgmap v992: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 833 op/s; 30691/353493 objects misplaced (8.682%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:27.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:26 smithi055 ceph-mon[40587]: 11.7 scrub ok 2024-03-26T21:49:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:26 smithi055 ceph-mon[29406]: pgmap v992: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 833 op/s; 30691/353493 objects misplaced (8.682%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:27.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:26 smithi055 ceph-mon[29406]: 11.7 scrub ok 2024-03-26T21:49:27.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:27.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:26 smithi067 ceph-mon[33346]: pgmap v992: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 833 op/s; 30691/353493 objects misplaced (8.682%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:26 smithi067 ceph-mon[33346]: 11.7 scrub ok 2024-03-26T21:49:27.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:27.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:27.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:27.554 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:27.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:27.688 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:27.790 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.842 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:27.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:27.943 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:27.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:27.992 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:27.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:28.093 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:28.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:28.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.277 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:28.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:28.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:28.414 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:28.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:28.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:28.684 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:28.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:28.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:28.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:28.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:28.966 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.6 primary-affinity to 0 (802) 2024-03-26T21:49:29.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:29.093 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:29.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:29.224 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: pgmap v993: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 848 op/s; 30595/349728 objects misplaced (8.748%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1020329659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:28 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:49:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: pgmap v993: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 848 op/s; 30595/349728 objects misplaced (8.748%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1020329659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: pgmap v993: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 7.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 848 op/s; 30595/349728 objects misplaced (8.748%); 27 MiB/s, 438 objects/s recovering 2024-03-26T21:49:29.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1020329659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:28 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-03-26T21:49:29.326 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:29.356 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:29.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:29.488 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:29.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:29.618 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:29.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:29.753 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:29.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:29.887 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:29.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125093 11.1 11.2985e261 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.92727528100000001, 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92730598399999997, 2024-03-26T21:49:29.988 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125093 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.989 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066425+0000", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.127e-06 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066431+0000", 2024-03-26T21:49:29.990 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.348e-06 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066446+0000", 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.467e-05 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066454+0000", 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8830000000000006e-06 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.991 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066540+0000", 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6306999999999999e-05 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066550+0000", 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7520000000000008e-06 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125122 11.1 11.6b627e01 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:29.992 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.85992195000000005, 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.86002671900000005, 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125122 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:29.993 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133778+0000", 2024-03-26T21:49:29.994 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1009999999999999e-06 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133783+0000", 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0030000000000002e-06 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133793+0000", 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9289999999999993e-06 2024-03-26T21:49:29.995 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133799+0000", 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0639999999999999e-06 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133870+0000", 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1194999999999999e-05 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:29.996 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133883+0000", 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2337999999999999e-05 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125044 11.1 11.669e2391 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.98619148599999995, 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98636089800000004, 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:29.997 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125044 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.998 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007509+0000", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.35e-06 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007513+0000", 2024-03-26T21:49:29.999 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3080000000000001e-06 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007521+0000", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6450000000000005e-06 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007527+0000", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.947e-06 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.008029+0000", 2024-03-26T21:49:30.000 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00050181200000000003 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.008040+0000", 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.092e-05 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125046 11.1 11.967b4c01 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.98201968900000003, 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.98225327100000004, 2024-03-26T21:49:30.001 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125046 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.002 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.003 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011681+0000", 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.477e-06 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.004 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011686+0000", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2290000000000002e-06 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011696+0000", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0325e-05 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011705+0000", 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0600000000000008e-06 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.005 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.012054+0000", 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00034953000000000002 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.012064+0000", 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.027e-05 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.006 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125100 11.1 11.9d323fa1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.91211908900000005, 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91241803899999996, 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125100 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.007 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.008 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081582+0000", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8530000000000001e-06 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081588+0000", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5139999999999998e-06 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081602+0000", 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3356000000000001e-05 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.009 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081610+0000", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6959999999999994e-06 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.082340+0000", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00072962600000000004 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.082353+0000", 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2415000000000001e-05 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.010 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125055 11.1 11.697eb611 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.97316098799999995, 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97352315599999995, 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.011 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125055 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.012 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020540+0000", 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.508e-06 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020545+0000", 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1680000000000003e-06 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020556+0000", 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.155e-05 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.013 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020565+0000", 2024-03-26T21:49:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4090000000000003e-06 2024-03-26T21:49:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.014 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020639+0000", 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3950999999999997e-05 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020649+0000", 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9420000000000007e-06 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.015 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125023 11.1 11.9d3c19c1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 1.0170572360000001, 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0174824579999999, 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125023 2024-03-26T21:49:30.016 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:49:30.017 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1399999999999997e-07 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976645+0000", 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9809999999999999e-06 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976651+0000", 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0689999999999998e-06 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.018 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976657+0000", 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8810000000000001e-06 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976915+0000", 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025755500000000002 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976923+0000", 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9950000000000005e-06 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.019 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125061 11.1 11.c29100e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.968402337, 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96893789900000005, 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125061 2024-03-26T21:49:30.020 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.021 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025298+0000", 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.299e-06 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025304+0000", 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3339999999999998e-06 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025314+0000", 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3579999999999992e-06 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.022 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025322+0000", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2609999999999993e-06 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025407+0000", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5407000000000004e-05 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025418+0000", 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0338e-05 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.023 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125063 11.1 11.d6dc3991 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.96690802399999998, 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96754273899999998, 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125063 2024-03-26T21:49:30.024 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026792+0000", 2024-03-26T21:49:30.025 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.249e-06 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026799+0000", 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8539999999999999e-06 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026815+0000", 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5523e-05 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.026 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026823+0000", 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3010000000000007e-06 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026907+0000", 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4260999999999998e-05 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026917+0000", 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9299999999999998e-06 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.027 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125111 11.1 11.63c65531 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 0.89877215799999999, 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.89948295899999997, 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "waiting for peered", 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125111 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:49:30.028 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094928+0000", 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8400000000000004e-07 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.029 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094932+0000", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0640000000000004e-06 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094940+0000", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4470000000000006e-06 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094947+0000", 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.443e-06 2024-03-26T21:49:30.030 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.097012+0000", 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020651699999999999 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.097022+0000", 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0214000000000001e-05 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:49:30.031 INFO:teuthology.orchestra.run.smithi055.stdout: "num_ops": 10 2024-03-26T21:49:30.032 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:49:30.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:30.167 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:30.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:30.225 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:30.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-26T21:49:30.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[40587]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T21:49:30.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:49:30.239 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:29 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:29 smithi067 ceph-mon[33346]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:29 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:29 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[29406]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:29 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:49:30.240 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:49:30.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:30.328 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:49:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:30.369 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:49:30.407 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:49:30.441 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:30.452 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:49:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:30.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:30.493 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:49:30.534 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:49:30.580 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:49:30.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:30.626 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:30.631 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:49:30.641 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:49:30.641 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:49:30.641 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:49:30.641 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:49:30.641 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:49:30.642 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:49:30.642 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:49:30.642 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:49:30.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:30.773 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:30.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:30.911 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:30.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:31.013 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:31.056 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:31.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:49:31.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: pgmap v995: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.03k op/s; 27856/340311 objects misplaced (8.185%); 33 MiB/s, 528 objects/s recovering 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/511331378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:49:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:30 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:49:31.245 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: pgmap v995: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.03k op/s; 27856/340311 objects misplaced (8.185%); 33 MiB/s, 528 objects/s recovering 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/511331378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:49:31.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: pgmap v995: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.03k op/s; 27856/340311 objects misplaced (8.185%); 33 MiB/s, 528 objects/s recovering 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/511331378' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:49:31.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:30 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:49:31.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:31.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:31.483 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:31.543 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:31.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:31.693 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:31.794 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:31.831 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:31.934 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:31.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:31.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:32.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:32.116 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:32.218 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.265 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:32.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:32.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:32 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:49:32.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:32 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:49:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:32 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:49:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:32 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:49:32.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:32.402 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:32 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:49:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:32 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:49:32.503 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.546 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:32.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:32.615 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:32.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:32.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:32.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:32.847 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:32.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:32.989 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:32.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[40587]: pgmap v997: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 646 B/s rd, 1.9 MiB/s wr, 669 op/s; 27720/338952 objects misplaced (8.178%); 24 MiB/s, 375 objects/s recovering 2024-03-26T21:49:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:33.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[29406]: pgmap v997: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 646 B/s rd, 1.9 MiB/s wr, 669 op/s; 27720/338952 objects misplaced (8.178%); 24 MiB/s, 375 objects/s recovering 2024-03-26T21:49:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:33.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:33 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:33.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:33.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:33.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:33.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:33.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:33.413 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:33 smithi067 ceph-mon[33346]: pgmap v997: 148 pgs: 2 peering, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 646 B/s rd, 1.9 MiB/s wr, 669 op/s; 27720/338952 objects misplaced (8.178%); 24 MiB/s, 375 objects/s recovering 2024-03-26T21:49:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:33 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:33 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:33.516 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.558 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:33.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:33.659 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.701 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:33.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:33.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:33.835 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:33.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:33.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:34.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:34.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:34.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.254 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:34.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:34 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:49:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:34 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:49:34.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:34 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:49:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:34 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:49:34.355 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:34.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:34 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:49:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:34 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:49:34.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:34.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:34.635 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:34.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:49:34.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:34.744 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:49:34.744 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:49:34.745 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T21:49:34.745 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T21:49:34.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:49:34.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:34.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:34.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:34.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:35.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:35 smithi055 ceph-mon[40587]: pgmap v998: 148 pgs: 2 peering, 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 632 KiB/s wr, 1.02k op/s; 25111/329457 objects misplaced (7.622%); 35 MiB/s, 558 objects/s recovering 2024-03-26T21:49:35.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:35 smithi055 ceph-mon[40587]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:49:35.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:35 smithi055 ceph-mon[29406]: pgmap v998: 148 pgs: 2 peering, 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 632 KiB/s wr, 1.02k op/s; 25111/329457 objects misplaced (7.622%); 35 MiB/s, 558 objects/s recovering 2024-03-26T21:49:35.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:35 smithi055 ceph-mon[29406]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:49:35.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:35.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:35.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:35.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:35 smithi067 ceph-mon[33346]: pgmap v998: 148 pgs: 2 peering, 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 140 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 632 KiB/s wr, 1.02k op/s; 25111/329457 objects misplaced (7.622%); 35 MiB/s, 558 objects/s recovering 2024-03-26T21:49:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:35 smithi067 ceph-mon[33346]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T21:49:35.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:35.538 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:35.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:35.676 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:35.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:35.820 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:35.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:35.971 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:35.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:36.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:36.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:36.134 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:49:36.135 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:49:36.187 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:36 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:49:36.187 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:36 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:49:36.187 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:36 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:49:36.187 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:36 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:49:36.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:36.270 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:36.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:36 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:49:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:36 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:49:36.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:36.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:36.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.826 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:36.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:36.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:36.963 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:36.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:37.056 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:49:37.064 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:37.115 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:37.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:37.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[29406]: pgmap v999: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 875 op/s; 24034/328725 objects misplaced (7.311%); 42 MiB/s, 674 objects/s recovering 2024-03-26T21:49:37.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T21:49:37.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4019142416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.267 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[40587]: pgmap v999: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 875 op/s; 24034/328725 objects misplaced (7.311%); 42 MiB/s, 674 objects/s recovering 2024-03-26T21:49:37.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T21:49:37.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4019142416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:37 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:49:37.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:37 smithi067 ceph-mon[33346]: pgmap v999: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 875 op/s; 24034/328725 objects misplaced (7.311%); 42 MiB/s, 674 objects/s recovering 2024-03-26T21:49:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:37 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T21:49:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4019142416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:37 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:49:37.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:37.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:37.635 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.680 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:37.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:37.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:37.819 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:37.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:37.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:37.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.000 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:38.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:38.056 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:38.077 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:38.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.168 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:38.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:38.207 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:49:38.207 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:49:38.207 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:38.207 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[29406]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T21:49:38.207 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:49:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:49:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:38.208 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-mon[40587]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T21:49:38.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:38.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:38.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:38 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:49:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:38 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:49:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:38 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:49:38.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:38 smithi067 ceph-mon[33346]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T21:49:38.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:38.580 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:38.682 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:38.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:38.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:38.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.005 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:39.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:39.065 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:39.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:39.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:39.250 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:39.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:39.343 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:39.343 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[40587]: pgmap v1001: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 0 B/s wr, 584 op/s; 23929/326280 objects misplaced (7.334%); 27 MiB/s, 433 objects/s recovering 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[40587]: 4.8 deep-scrub starts 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[40587]: 4.8 deep-scrub ok 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[29406]: pgmap v1001: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 0 B/s wr, 584 op/s; 23929/326280 objects misplaced (7.334%); 27 MiB/s, 433 objects/s recovering 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[29406]: 4.8 deep-scrub starts 2024-03-26T21:49:39.344 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:39 smithi055 ceph-mon[29406]: 4.8 deep-scrub ok 2024-03-26T21:49:39.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:39.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:39 smithi067 ceph-mon[33346]: pgmap v1001: 148 pgs: 4 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 142 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 255 B/s rd, 0 B/s wr, 584 op/s; 23929/326280 objects misplaced (7.334%); 27 MiB/s, 433 objects/s recovering 2024-03-26T21:49:39.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:39 smithi067 ceph-mon[33346]: 4.8 deep-scrub starts 2024-03-26T21:49:39.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:39 smithi067 ceph-mon[33346]: 4.8 deep-scrub ok 2024-03-26T21:49:39.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.429 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:39.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:39.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:39.567 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:39.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:39.813 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:39.857 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:39.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:39.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:40.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:49:40.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:40.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:40.237 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:40.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:40 smithi055 ceph-mon[40587]: Health check failed: Degraded data redundancy: 169/315834 objects degraded (0.054%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:49:40.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:40 smithi055 ceph-mon[29406]: Health check failed: Degraded data redundancy: 169/315834 objects degraded (0.054%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:49:40.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:40.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:40 smithi067 ceph-mon[33346]: Health check failed: Degraded data redundancy: 169/315834 objects degraded (0.054%), 1 pg degraded (PG_DEGRADED) 2024-03-26T21:49:40.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:40.588 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:40.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:40.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:40.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:40.872 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:40.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:40.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:41.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:41.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:41.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:41.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:41.433 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:41.468 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[40587]: pgmap v1002: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 834 op/s; 169/315834 objects degraded (0.054%); 20648/315834 objects misplaced (6.538%); 39 MiB/s, 629 objects/s recovering 2024-03-26T21:49:41.468 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:41.468 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:41.468 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[29406]: pgmap v1002: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 834 op/s; 169/315834 objects degraded (0.054%); 20648/315834 objects misplaced (6.538%); 39 MiB/s, 629 objects/s recovering 2024-03-26T21:49:41.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:41.469 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:41 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:41 smithi067 ceph-mon[33346]: pgmap v1002: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 834 op/s; 169/315834 objects degraded (0.054%); 20648/315834 objects misplaced (6.538%); 39 MiB/s, 629 objects/s recovering 2024-03-26T21:49:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:41 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:41.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:41 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:41.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:41.569 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:41.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:41.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:41.811 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:41.849 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:41.951 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:41.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:41.999 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:41.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:42.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:42.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:42.279 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:42.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:42 smithi067 ceph-mon[33346]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T21:49:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:42 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:49:42.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:42 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:49:42.514 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[40587]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T21:49:42.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:49:42.515 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:49:42.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[29406]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T21:49:42.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:49:42.515 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:42 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:49:42.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:42.654 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:42.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:42.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:42.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:42.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:43.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:43.102 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:43.204 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:43.240 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:43.342 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.380 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:43.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:43.481 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:43 smithi067 ceph-mon[33346]: pgmap v1004: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s; 316/314154 objects degraded (0.101%); 20378/314154 objects misplaced (6.487%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:49:43.517 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:43.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:43 smithi055 ceph-mon[40587]: pgmap v1004: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s; 316/314154 objects degraded (0.101%); 20378/314154 objects misplaced (6.487%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:49:43.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:43 smithi055 ceph-mon[29406]: pgmap v1004: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 638 op/s; 316/314154 objects degraded (0.101%); 20378/314154 objects misplaced (6.487%); 31 MiB/s, 491 objects/s recovering 2024-03-26T21:49:43.620 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.646 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:43.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:43.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:43.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:43.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:43.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:44.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:44.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:44.192 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:44.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:44.322 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:44.424 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:44.451 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:44 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:49:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:44 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:49:44.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:44.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:44 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:49:44.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:44 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:49:44.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:44 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:49:44.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:44 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:49:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:44.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:44.680 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.710 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:44.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:44.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:44.837 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:44.939 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:44.965 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:44.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:45.067 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:45.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:45.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.221 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:45.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:45 smithi055 ceph-mon[40587]: pgmap v1005: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 502/303423 objects degraded (0.165%); 17987/303423 objects misplaced (5.928%); 37 MiB/s, 598 objects/s recovering 2024-03-26T21:49:45.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:45 smithi055 ceph-mon[40587]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T21:49:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:45 smithi055 ceph-mon[29406]: pgmap v1005: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 502/303423 objects degraded (0.165%); 17987/303423 objects misplaced (5.928%); 37 MiB/s, 598 objects/s recovering 2024-03-26T21:49:45.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:45 smithi055 ceph-mon[29406]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T21:49:45.321 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:45.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:45.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:45 smithi067 ceph-mon[33346]: pgmap v1005: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 142 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 502/303423 objects degraded (0.165%); 17987/303423 objects misplaced (5.928%); 37 MiB/s, 598 objects/s recovering 2024-03-26T21:49:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:45 smithi067 ceph-mon[33346]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T21:49:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:45.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:45.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:45.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:45.666 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:49:45.677 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:49:45.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:45.778 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:45.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:46.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:46.054 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:46.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:46.192 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:46.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.319 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:46.324 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:46 smithi055 ceph-mon[29406]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:49:46.324 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:46 smithi055 ceph-mon[40587]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:49:46.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:46.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:46.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:46 smithi067 ceph-mon[33346]: from='client.15897 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:49:46.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:46 smithi067 ceph-mon[33346]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T21:49:46.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.584 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:46.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:46.723 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:46.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:46 smithi055 ceph-mon[40587]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T21:49:46.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:46 smithi055 ceph-mon[29406]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T21:49:46.825 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:46.857 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:46.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:46.995 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:46.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:47.097 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:47.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:47.236 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:47.275 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:47.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:47.416 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:47.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:47.546 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[40587]: pgmap v1008: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 738 op/s; 502/302556 objects degraded (0.166%); 17329/302556 objects misplaced (5.728%); 27 MiB/s, 439 objects/s recovering 2024-03-26T21:49:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 502/302556 objects degraded (0.166%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:47.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[29406]: pgmap v1008: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 738 op/s; 502/302556 objects degraded (0.166%); 17329/302556 objects misplaced (5.728%); 27 MiB/s, 439 objects/s recovering 2024-03-26T21:49:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 502/302556 objects degraded (0.166%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:47.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:47 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:47.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:47.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:47.686 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:47 smithi067 ceph-mon[33346]: pgmap v1008: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 738 op/s; 502/302556 objects degraded (0.166%); 17329/302556 objects misplaced (5.728%); 27 MiB/s, 439 objects/s recovering 2024-03-26T21:49:47.686 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 502/302556 objects degraded (0.166%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:47.687 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:47 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:47.687 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:47 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:47.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:47.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:47.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:47.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:47.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:48.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.102 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:48.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:48.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:48.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:48.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:48.379 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:48.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:48.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:48.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:48 smithi067 ceph-mon[33346]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T21:49:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:48.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:48.744 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:48.789 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:48.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[40587]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T21:49:48.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:48.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[29406]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T21:49:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:48.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:49:48.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:48.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:48.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:49.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:49.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:49.171 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:49.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:49.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:49.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:49.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:49.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.475 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:49.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:49.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:49.615 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:49.717 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:49 smithi067 ceph-mon[33346]: pgmap v1010: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 757 op/s; 634/300522 objects degraded (0.211%); 17065/300522 objects misplaced (5.678%); 28 MiB/s, 440 objects/s recovering 2024-03-26T21:49:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:49 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:49:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:49 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:49:49.757 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:49.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:49.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[40587]: pgmap v1010: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 757 op/s; 634/300522 objects degraded (0.211%); 17065/300522 objects misplaced (5.678%); 28 MiB/s, 440 objects/s recovering 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[29406]: pgmap v1010: 148 pgs: 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 142 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 757 op/s; 634/300522 objects degraded (0.211%); 17065/300522 objects misplaced (5.678%); 28 MiB/s, 440 objects/s recovering 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:49:49.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:49 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:49:49.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:49.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:49.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:50.030 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:50.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:50.168 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:50.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:50.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:50.442 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:50.544 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:50.585 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:50.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.722 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:50.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:50 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:50 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:50.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:50 smithi067 ceph-mon[33346]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T21:49:50.774 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:49:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:50.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[40587]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T21:49:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:50.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:50 smithi055 ceph-mon[29406]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T21:49:50.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:50.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:50.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:50.970 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:51.027 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:51.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:51.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:51.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:51.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:51.403 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:51.444 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:51.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:51.673 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: pgmap v1012: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 618 op/s; 700/292260 objects degraded (0.240%); 14961/292260 objects misplaced (5.119%); 22 MiB/s, 352 objects/s recovering 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: pgmap v1012: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 618 op/s; 700/292260 objects degraded (0.240%); 14961/292260 objects misplaced (5.119%); 22 MiB/s, 352 objects/s recovering 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:51.674 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:51.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T21:49:51.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:51.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:49:51.675 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:51 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:49:51.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:51.712 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: pgmap v1012: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 0 B/s wr, 618 op/s; 700/292260 objects degraded (0.240%); 14961/292260 objects misplaced (5.119%); 22 MiB/s, 352 objects/s recovering 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:49:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:51 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:49:51.814 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.854 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:51.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:51.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:51.980 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:51.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:52.082 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:52.122 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:52.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.263 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:52.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:52.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:52.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:52.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:52.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:52.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 700/292260 objects degraded (0.240%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:52.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:52.802 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:52.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 700/292260 objects degraded (0.240%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:52.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 700/292260 objects degraded (0.240%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:52.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:52.944 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:52.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:53.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:53.086 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:53.188 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.227 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:53.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:53.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:53.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:53.461 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:53.496 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:53.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:53.639 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:53 smithi067 ceph-mon[33346]: pgmap v1014: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 855/290586 objects degraded (0.294%); 14706/290586 objects misplaced (5.061%); 21 MiB/s, 343 objects/s recovering 2024-03-26T21:49:53.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.778 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:53.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:53.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:53 smithi055 ceph-mon[40587]: pgmap v1014: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 855/290586 objects degraded (0.294%); 14706/290586 objects misplaced (5.061%); 21 MiB/s, 343 objects/s recovering 2024-03-26T21:49:53.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:53 smithi055 ceph-mon[29406]: pgmap v1014: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 855/290586 objects degraded (0.294%); 14706/290586 objects misplaced (5.061%); 21 MiB/s, 343 objects/s recovering 2024-03-26T21:49:53.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:53.904 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:53.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:54.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:54.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:54.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:54.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:54.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:54.441 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:54.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:54.580 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:54.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:54.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:54.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:54.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:55.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:55.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:55.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:55.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:55.401 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:55.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:55.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:55 smithi067 ceph-mon[33346]: pgmap v1015: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s; 1044/278931 objects degraded (0.374%); 13141/278931 objects misplaced (4.711%); 30 MiB/s, 481 objects/s recovering 2024-03-26T21:49:55.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:55.807 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:55.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:55 smithi055 ceph-mon[40587]: pgmap v1015: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s; 1044/278931 objects degraded (0.374%); 13141/278931 objects misplaced (4.711%); 30 MiB/s, 481 objects/s recovering 2024-03-26T21:49:55.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:55 smithi055 ceph-mon[29406]: pgmap v1015: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.12k op/s; 1044/278931 objects degraded (0.374%); 13141/278931 objects misplaced (4.711%); 30 MiB/s, 481 objects/s recovering 2024-03-26T21:49:55.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:55.938 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:55.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:56.040 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.084 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:56.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:56.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:56.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:56.319 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:56.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:56.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:56.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:56.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:56 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:49:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:56 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:49:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:56 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:56 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:56.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:56 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:56.765 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:56.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:49:56.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:49:56.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:56.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:56.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:56 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:49:56.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:56.895 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:56.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:56.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:57.023 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:57.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:57.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:57.276 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:57.378 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.405 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:57.409 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[29406]: pgmap v1016: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 937 op/s; 1044/278052 objects degraded (0.375%); 13141/278052 objects misplaced (4.726%); 24 MiB/s, 385 objects/s recovering 2024-03-26T21:49:57.409 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[29406]: osdmap e476: 8 total, 8 up, 7 in 2024-03-26T21:49:57.409 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1044/278052 objects degraded (0.375%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:57.507 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.546 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:57.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:49:57.690 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:49:57.698 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:57 smithi067 ceph-mon[33346]: pgmap v1016: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 937 op/s; 1044/278052 objects degraded (0.375%); 13141/278052 objects misplaced (4.726%); 24 MiB/s, 385 objects/s recovering 2024-03-26T21:49:57.698 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:57 smithi067 ceph-mon[33346]: osdmap e476: 8 total, 8 up, 7 in 2024-03-26T21:49:57.699 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1044/278052 objects degraded (0.375%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:57.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:57.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[40587]: pgmap v1016: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 937 op/s; 1044/278052 objects degraded (0.375%); 13141/278052 objects misplaced (4.726%); 24 MiB/s, 385 objects/s recovering 2024-03-26T21:49:57.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[40587]: osdmap e476: 8 total, 8 up, 7 in 2024-03-26T21:49:57.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1044/278052 objects degraded (0.375%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:49:57.828 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:57.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:57.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:57.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:49:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:49:58.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:49:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:49:58.207 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:58.237 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:58.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:58.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:58.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:49:58.509 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:49:58.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:58.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:58.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:58.737 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:58 smithi067 ceph-mon[33346]: osdmap e477: 8 total, 8 up, 7 in 2024-03-26T21:49:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:58 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:49:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:58 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:49:58.772 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:58.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:58.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[29406]: osdmap e477: 8 total, 8 up, 7 in 2024-03-26T21:49:58.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:49:58.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:49:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[40587]: osdmap e477: 8 total, 8 up, 7 in 2024-03-26T21:49:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:49:58.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:49:58.874 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:58.901 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:58.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:59.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:59.140 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:49:59.180 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:49:59.282 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:49:59.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:49:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:49:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:49:59.320 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:49:59.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:49:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:49:59.557 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:49:59.600 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:49:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:49:59.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:59.739 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:59 smithi067 ceph-mon[33346]: pgmap v1019: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 694 B/s rd, 0 B/s wr, 756 op/s; 1232/275526 objects degraded (0.447%); 12843/275526 objects misplaced (4.661%); 12 MiB/s, 185 objects/s recovering 2024-03-26T21:49:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:59 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:49:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:49:59 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:49:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[29406]: pgmap v1019: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 694 B/s rd, 0 B/s wr, 756 op/s; 1232/275526 objects degraded (0.447%); 12843/275526 objects misplaced (4.661%); 12 MiB/s, 185 objects/s recovering 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[40587]: pgmap v1019: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 144 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 694 B/s rd, 0 B/s wr, 756 op/s; 1232/275526 objects degraded (0.447%); 12843/275526 objects misplaced (4.661%); 12 MiB/s, 185 objects/s recovering 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:49:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:49:59 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:49:59.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:49:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:49:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:49:59.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:49:59.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.016 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:00.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:00.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:00.138 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:00.139 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.161 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:00.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:00.171 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:00.176 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:00.177 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:00.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:00.302 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:00.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:00.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:00.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:00.736 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: osdmap e478: 8 total, 8 up, 7 in 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3638331515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: Health detail: HEALTH_WARN Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: [WRN] PG_DEGRADED: Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: pg 11.2 is active+recovering+undersized+degraded+remapped, acting [3,0] 2024-03-26T21:50:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:00 smithi067 ceph-mon[33346]: pg 11.d is active+recovering+undersized+degraded+remapped, acting [4,6] 2024-03-26T21:50:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: osdmap e478: 8 total, 8 up, 7 in 2024-03-26T21:50:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3638331515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:00.779 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: Health detail: HEALTH_WARN Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: [WRN] PG_DEGRADED: Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: pg 11.2 is active+recovering+undersized+degraded+remapped, acting [3,0] 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[29406]: pg 11.d is active+recovering+undersized+degraded+remapped, acting [4,6] 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: osdmap e478: 8 total, 8 up, 7 in 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3638331515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: Health detail: HEALTH_WARN Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: [WRN] PG_DEGRADED: Degraded data redundancy: 1232/275526 objects degraded (0.447%), 2 pgs degraded 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: pg 11.2 is active+recovering+undersized+degraded+remapped, acting [3,0] 2024-03-26T21:50:00.780 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:00 smithi055 ceph-mon[40587]: pg 11.d is active+recovering+undersized+degraded+remapped, acting [4,6] 2024-03-26T21:50:00.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:00.874 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:00.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:00.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:01.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:01.152 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:01.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:01.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:01.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:01.432 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:01.534 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:01.574 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:01.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:01.713 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:01.721 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[40587]: pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 707 op/s; 1337/266196 objects degraded (0.502%); 11302/266196 objects misplaced (4.246%); 15 MiB/s, 234 objects/s recovering 2024-03-26T21:50:01.721 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:50:01.721 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:50:01.721 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[29406]: pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 707 op/s; 1337/266196 objects degraded (0.502%); 11302/266196 objects misplaced (4.246%); 15 MiB/s, 234 objects/s recovering 2024-03-26T21:50:01.721 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:50:01.722 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:01 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:50:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:01 smithi067 ceph-mon[33346]: pgmap v1021: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 707 op/s; 1337/266196 objects degraded (0.502%); 11302/266196 objects misplaced (4.246%); 15 MiB/s, 234 objects/s recovering 2024-03-26T21:50:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:01 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:50:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:01 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:50:01.816 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.857 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:01.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:01.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:01.996 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:01.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:02.098 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:02.138 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:02.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:02.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:02.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:02.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:02.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:02.544 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:02.617 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1006,"stamp":"2024-03-26T21:49:44.197576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6596991424,"num_objects":100852,"num_object_clones":0,"num_object_copies":302556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":17329,"num_objects_unfound":0,"num_objects_dirty":100852,"num_whiteouts":0,"num_read":2890,"num_read_kb":2738,"num_write":137773,"num_write_kb":7594078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21972,"num_bytes_recovered":1417356506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75831,"ondisk_log_size":75831,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23708296,"kb_used_data":18306440,"kb_used_omap":18,"kb_used_meta":5401773,"kb_avail":632507768,"statfs":{"total":671965249536,"available":647687954432,"internally_reserved":0,"allocated":18745794560,"data_stored":18742817160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19289,"internal_metadata":5531415719},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536150016,"num_objects":-8181,"num_object_clones":0,"num_object_copies":-24543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":-6600,"num_objects_unfound":0,"num_objects_dirty":-8181,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5325,"num_bytes_recovered":348979200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.148871"},"pg_stats":[{"pgid":"11.3","version":"470'8580","reported_seq":8674,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.486154+0000","last_change":"2024-03-26T21:49:29.997467+0000","last_active":"2024-03-26T21:49:43.486154+0000","last_peered":"2024-03-26T21:49:43.486154+0000","last_clean":"2024-03-26T21:49:43.486154+0000","last_became_active":"2024-03-26T21:49:29.980238+0000","last_became_peered":"2024-03-26T21:49:29.980238+0000","last_unstale":"2024-03-26T21:49:43.486154+0000","last_undegraded":"2024-03-26T21:49:43.486154+0000","last_fullsized":"2024-03-26T21:49:43.486154+0000","mapping_epoch":467,"log_start":"449'3877","ondisk_log_start":"449'3877","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":99,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8580,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'35","reported_seq":674,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179081+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:49:41.179081+0000","last_peered":"2024-03-26T21:49:41.179081+0000","last_clean":"2024-03-26T21:49:41.179081+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:49:41.179081+0000","last_undegraded":"2024-03-26T21:49:41.179081+0000","last_fullsized":"2024-03-26T21:49:41.179081+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177169+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:49:41.177169+0000","last_peered":"2024-03-26T21:49:41.177169+0000","last_clean":"2024-03-26T21:49:41.177169+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:49:41.177169+0000","last_undegraded":"2024-03-26T21:49:41.177169+0000","last_fullsized":"2024-03-26T21:49:41.177169+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"462'12","reported_seq":683,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169074+0000","last_change":"2024-03-26T21:48:59.006658+0000","last_active":"2024-03-26T21:49:41.169074+0000","last_peered":"2024-03-26T21:49:41.169074+0000","last_clean":"2024-03-26T21:49:41.169074+0000","last_became_active":"2024-03-26T21:48:58.995625+0000","last_became_peered":"2024-03-26T21:48:58.995625+0000","last_unstale":"2024-03-26T21:49:41.169074+0000","last_undegraded":"2024-03-26T21:49:41.169074+0000","last_fullsized":"2024-03-26T21:49:41.169074+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":584,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179391+0000","last_change":"2024-03-26T21:48:58.976822+0000","last_active":"2024-03-26T21:49:41.179391+0000","last_peered":"2024-03-26T21:49:41.179391+0000","last_clean":"2024-03-26T21:49:41.179391+0000","last_became_active":"2024-03-26T21:48:58.976172+0000","last_became_peered":"2024-03-26T21:48:58.976172+0000","last_unstale":"2024-03-26T21:49:41.179391+0000","last_undegraded":"2024-03-26T21:49:41.179391+0000","last_fullsized":"2024-03-26T21:49:41.179391+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"470'8583","reported_seq":10200,"reported_epoch":470,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:49:43.243710+0000","last_change":"2024-03-26T21:49:33.951482+0000","last_active":"2024-03-26T21:49:43.243710+0000","last_peered":"2024-03-26T21:49:43.243710+0000","last_clean":"2024-03-26T21:48:57.900582+0000","last_became_active":"2024-03-26T21:49:00.154091+0000","last_became_peered":"2024-03-26T21:49:00.154091+0000","last_unstale":"2024-03-26T21:49:43.243710+0000","last_undegraded":"2024-03-26T21:49:33.947342+0000","last_fullsized":"2024-03-26T21:48:59.963248+0000","mapping_epoch":462,"log_start":"449'3999","ondisk_log_start":"449'3999","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":849,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416481280,"num_objects":6355,"num_object_clones":0,"num_object_copies":19065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":355,"num_objects_misplaced":5229,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":760,"num_bytes_recovered":49807360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":5229},{"shards":"0,3","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":670,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.182500+0000","last_change":"2024-03-26T21:48:59.000246+0000","last_active":"2024-03-26T21:49:41.182500+0000","last_peered":"2024-03-26T21:49:41.182500+0000","last_clean":"2024-03-26T21:49:41.182500+0000","last_became_active":"2024-03-26T21:48:58.980284+0000","last_became_peered":"2024-03-26T21:48:58.980284+0000","last_unstale":"2024-03-26T21:49:41.182500+0000","last_undegraded":"2024-03-26T21:49:41.182500+0000","last_fullsized":"2024-03-26T21:49:41.182500+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":342,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177827+0000","last_change":"2024-03-26T21:49:29.974892+0000","last_active":"2024-03-26T21:49:41.177827+0000","last_peered":"2024-03-26T21:49:41.177827+0000","last_clean":"2024-03-26T21:49:41.177827+0000","last_became_active":"2024-03-26T21:49:29.974562+0000","last_became_peered":"2024-03-26T21:49:29.974562+0000","last_unstale":"2024-03-26T21:49:41.177827+0000","last_undegraded":"2024-03-26T21:49:41.177827+0000","last_fullsized":"2024-03-26T21:49:41.177827+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":911,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.181080+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:49:41.181080+0000","last_peered":"2024-03-26T21:49:41.181080+0000","last_clean":"2024-03-26T21:49:41.181080+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:49:41.181080+0000","last_undegraded":"2024-03-26T21:49:41.181080+0000","last_fullsized":"2024-03-26T21:49:41.181080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":708,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176649+0000","last_change":"2024-03-26T21:48:58.974425+0000","last_active":"2024-03-26T21:49:41.176649+0000","last_peered":"2024-03-26T21:49:41.176649+0000","last_clean":"2024-03-26T21:49:41.176649+0000","last_became_active":"2024-03-26T21:48:58.974080+0000","last_became_peered":"2024-03-26T21:48:58.974080+0000","last_unstale":"2024-03-26T21:49:41.176649+0000","last_undegraded":"2024-03-26T21:49:41.176649+0000","last_fullsized":"2024-03-26T21:49:41.176649+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"470'8468","reported_seq":8558,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.224617+0000","last_change":"2024-03-26T21:49:29.994417+0000","last_active":"2024-03-26T21:49:43.224617+0000","last_peered":"2024-03-26T21:49:43.224617+0000","last_clean":"2024-03-26T21:49:43.224617+0000","last_became_active":"2024-03-26T21:49:29.979423+0000","last_became_peered":"2024-03-26T21:49:29.979423+0000","last_unstale":"2024-03-26T21:49:43.224617+0000","last_undegraded":"2024-03-26T21:49:43.224617+0000","last_fullsized":"2024-03-26T21:49:43.224617+0000","mapping_epoch":467,"log_start":"449'3866","ondisk_log_start":"449'3866","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406585344,"num_objects":6204,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":469504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":619,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179484+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:49:41.179484+0000","last_peered":"2024-03-26T21:49:41.179484+0000","last_clean":"2024-03-26T21:49:41.179484+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:49:41.179484+0000","last_undegraded":"2024-03-26T21:49:41.179484+0000","last_fullsized":"2024-03-26T21:49:41.179484+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":593,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177857+0000","last_change":"2024-03-26T21:48:58.979871+0000","last_active":"2024-03-26T21:49:41.177857+0000","last_peered":"2024-03-26T21:49:41.177857+0000","last_clean":"2024-03-26T21:49:41.177857+0000","last_became_active":"2024-03-26T21:48:58.979258+0000","last_became_peered":"2024-03-26T21:48:58.979258+0000","last_unstale":"2024-03-26T21:49:41.177857+0000","last_undegraded":"2024-03-26T21:49:41.177857+0000","last_fullsized":"2024-03-26T21:49:41.177857+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.750677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:46.291329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":893,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174811+0000","last_change":"2024-03-26T21:48:16.730412+0000","last_active":"2024-03-26T21:49:41.174811+0000","last_peered":"2024-03-26T21:49:41.174811+0000","last_clean":"2024-03-26T21:49:41.174811+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:49:41.174811+0000","last_undegraded":"2024-03-26T21:49:41.174811+0000","last_fullsized":"2024-03-26T21:49:41.174811+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.730303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.730303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176480+0000","last_change":"2024-03-26T21:48:30.759102+0000","last_active":"2024-03-26T21:49:41.176480+0000","last_peered":"2024-03-26T21:49:41.176480+0000","last_clean":"2024-03-26T21:49:41.176480+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:49:41.176480+0000","last_undegraded":"2024-03-26T21:49:41.176480+0000","last_fullsized":"2024-03-26T21:49:41.176480+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.758986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.758986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"470'8492","reported_seq":8576,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.224922+0000","last_change":"2024-03-26T21:47:39.053277+0000","last_active":"2024-03-26T21:49:43.224922+0000","last_peered":"2024-03-26T21:49:43.224922+0000","last_clean":"2024-03-26T21:49:43.224922+0000","last_became_active":"2024-03-26T21:47:39.052942+0000","last_became_peered":"2024-03-26T21:47:39.052942+0000","last_unstale":"2024-03-26T21:49:43.224922+0000","last_undegraded":"2024-03-26T21:49:43.224922+0000","last_fullsized":"2024-03-26T21:49:43.224922+0000","mapping_epoch":433,"log_start":"450'4152","ondisk_log_start":"450'4152","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405405696,"num_objects":6186,"num_object_clones":0,"num_object_copies":18558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1048,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.186485+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:49:41.186485+0000","last_peered":"2024-03-26T21:49:41.186485+0000","last_clean":"2024-03-26T21:49:41.186485+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:49:41.186485+0000","last_undegraded":"2024-03-26T21:49:41.186485+0000","last_fullsized":"2024-03-26T21:49:41.186485+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179332+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:49:41.179332+0000","last_peered":"2024-03-26T21:49:41.179332+0000","last_clean":"2024-03-26T21:49:41.179332+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:49:41.179332+0000","last_undegraded":"2024-03-26T21:49:41.179332+0000","last_fullsized":"2024-03-26T21:49:41.179332+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":745,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179597+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:49:41.179597+0000","last_peered":"2024-03-26T21:49:41.179597+0000","last_clean":"2024-03-26T21:49:41.179597+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:49:41.179597+0000","last_undegraded":"2024-03-26T21:49:41.179597+0000","last_fullsized":"2024-03-26T21:49:41.179597+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":876,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174469+0000","last_change":"2024-03-26T21:48:19.935968+0000","last_active":"2024-03-26T21:49:41.174469+0000","last_peered":"2024-03-26T21:49:41.174469+0000","last_clean":"2024-03-26T21:49:41.174469+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:49:41.174469+0000","last_undegraded":"2024-03-26T21:49:41.174469+0000","last_fullsized":"2024-03-26T21:49:41.174469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.935613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.935613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.386876+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"470'8492","reported_seq":22113,"reported_epoch":470,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T21:49:43.491357+0000","last_change":"2024-03-26T21:49:00.072982+0000","last_active":"2024-03-26T21:49:43.491357+0000","last_peered":"2024-03-26T21:49:43.491357+0000","last_clean":"2024-03-26T21:48:57.962183+0000","last_became_active":"2024-03-26T21:49:00.062409+0000","last_became_peered":"2024-03-26T21:49:00.062409+0000","last_unstale":"2024-03-26T21:49:43.491357+0000","last_undegraded":"2024-03-26T21:49:43.491357+0000","last_fullsized":"2024-03-26T21:49:43.491357+0000","mapping_epoch":462,"log_start":"449'3791","ondisk_log_start":"449'3791","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411435008,"num_objects":6278,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6766,"num_bytes_recovered":443416576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,2],"avail_no_missing":["7","1","2"],"object_location_counts":[{"shards":"1,2,7","objects":6278}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":598,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178057+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:49:41.178057+0000","last_peered":"2024-03-26T21:49:41.178057+0000","last_clean":"2024-03-26T21:49:41.178057+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:49:41.178057+0000","last_undegraded":"2024-03-26T21:49:41.178057+0000","last_fullsized":"2024-03-26T21:49:41.178057+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":269,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179015+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:49:41.179015+0000","last_peered":"2024-03-26T21:49:41.179015+0000","last_clean":"2024-03-26T21:49:41.179015+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:49:41.179015+0000","last_undegraded":"2024-03-26T21:49:41.179015+0000","last_fullsized":"2024-03-26T21:49:41.179015+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177312+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:49:41.177312+0000","last_peered":"2024-03-26T21:49:41.177312+0000","last_clean":"2024-03-26T21:49:41.177312+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:49:41.177312+0000","last_undegraded":"2024-03-26T21:49:41.177312+0000","last_fullsized":"2024-03-26T21:49:41.177312+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":605,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178195+0000","last_change":"2024-03-26T21:48:58.977779+0000","last_active":"2024-03-26T21:49:41.178195+0000","last_peered":"2024-03-26T21:49:41.178195+0000","last_clean":"2024-03-26T21:49:41.178195+0000","last_became_active":"2024-03-26T21:48:58.977356+0000","last_became_peered":"2024-03-26T21:48:58.977356+0000","last_unstale":"2024-03-26T21:49:41.178195+0000","last_undegraded":"2024-03-26T21:49:41.178195+0000","last_fullsized":"2024-03-26T21:49:41.178195+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:28.651656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:48:28.651656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:58.077774+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"470'8405","reported_seq":8489,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.252498+0000","last_change":"2024-03-26T21:47:39.053658+0000","last_active":"2024-03-26T21:49:41.252498+0000","last_peered":"2024-03-26T21:49:41.252498+0000","last_clean":"2024-03-26T21:49:41.252498+0000","last_became_active":"2024-03-26T21:47:39.053244+0000","last_became_peered":"2024-03-26T21:47:39.053244+0000","last_unstale":"2024-03-26T21:49:41.252498+0000","last_undegraded":"2024-03-26T21:49:41.252498+0000","last_fullsized":"2024-03-26T21:49:41.252498+0000","mapping_epoch":433,"log_start":"448'3698","ondisk_log_start":"448'3698","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414646272,"num_objects":6327,"num_object_clones":0,"num_object_copies":18981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":471424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":296,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179411+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:49:41.179411+0000","last_peered":"2024-03-26T21:49:41.179411+0000","last_clean":"2024-03-26T21:49:41.179411+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:49:41.179411+0000","last_undegraded":"2024-03-26T21:49:41.179411+0000","last_fullsized":"2024-03-26T21:49:41.179411+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":637,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177896+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:49:41.177896+0000","last_peered":"2024-03-26T21:49:41.177896+0000","last_clean":"2024-03-26T21:49:41.177896+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:49:41.177896+0000","last_undegraded":"2024-03-26T21:49:41.177896+0000","last_fullsized":"2024-03-26T21:49:41.177896+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":896,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180446+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:49:41.180446+0000","last_peered":"2024-03-26T21:49:41.180446+0000","last_clean":"2024-03-26T21:49:41.180446+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:49:41.180446+0000","last_undegraded":"2024-03-26T21:49:41.180446+0000","last_fullsized":"2024-03-26T21:49:41.180446+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":808,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168988+0000","last_change":"2024-03-26T21:48:58.986678+0000","last_active":"2024-03-26T21:49:41.168988+0000","last_peered":"2024-03-26T21:49:41.168988+0000","last_clean":"2024-03-26T21:49:41.168988+0000","last_became_active":"2024-03-26T21:48:58.985775+0000","last_became_peered":"2024-03-26T21:48:58.985775+0000","last_unstale":"2024-03-26T21:49:41.168988+0000","last_undegraded":"2024-03-26T21:49:41.168988+0000","last_fullsized":"2024-03-26T21:49:41.168988+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"470'8383","reported_seq":10216,"reported_epoch":470,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:49:41.331441+0000","last_change":"2024-03-26T21:49:36.580498+0000","last_active":"2024-03-26T21:49:41.331441+0000","last_peered":"2024-03-26T21:49:41.331441+0000","last_clean":"2024-03-26T21:48:57.955814+0000","last_became_active":"2024-03-26T21:49:00.339243+0000","last_became_peered":"2024-03-26T21:49:00.339243+0000","last_unstale":"2024-03-26T21:49:41.331441+0000","last_undegraded":"2024-03-26T21:49:36.577298+0000","last_fullsized":"2024-03-26T21:48:59.964175+0000","mapping_epoch":462,"log_start":"448'3674","ondisk_log_start":"448'3674","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":3599,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418381848,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":147,"num_objects_misplaced":5262,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8383,"num_write_kb":472513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":55836672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":5262},{"shards":"4,6","objects":976}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1137,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.183536+0000","last_change":"2024-03-26T21:48:32.957198+0000","last_active":"2024-03-26T21:49:41.183536+0000","last_peered":"2024-03-26T21:49:41.183536+0000","last_clean":"2024-03-26T21:49:41.183536+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:49:41.183536+0000","last_undegraded":"2024-03-26T21:49:41.183536+0000","last_fullsized":"2024-03-26T21:49:41.183536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:48:32.957059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.957059+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":977,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177550+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:49:41.177550+0000","last_peered":"2024-03-26T21:49:41.177550+0000","last_clean":"2024-03-26T21:49:41.177550+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:49:41.177550+0000","last_undegraded":"2024-03-26T21:49:41.177550+0000","last_fullsized":"2024-03-26T21:49:41.177550+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":915,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174762+0000","last_change":"2024-03-26T21:49:27.705867+0000","last_active":"2024-03-26T21:49:41.174762+0000","last_peered":"2024-03-26T21:49:41.174762+0000","last_clean":"2024-03-26T21:49:41.174762+0000","last_became_active":"2024-03-26T21:48:58.979760+0000","last_became_peered":"2024-03-26T21:48:58.979760+0000","last_unstale":"2024-03-26T21:49:41.174762+0000","last_undegraded":"2024-03-26T21:49:41.174762+0000","last_fullsized":"2024-03-26T21:49:41.174762+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:27.705733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:49:27.705733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:33.766431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":889,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179569+0000","last_change":"2024-03-26T21:48:58.973098+0000","last_active":"2024-03-26T21:49:41.179569+0000","last_peered":"2024-03-26T21:49:41.179569+0000","last_clean":"2024-03-26T21:49:41.179569+0000","last_became_active":"2024-03-26T21:48:58.972760+0000","last_became_peered":"2024-03-26T21:48:58.972760+0000","last_unstale":"2024-03-26T21:49:41.179569+0000","last_undegraded":"2024-03-26T21:49:41.179569+0000","last_fullsized":"2024-03-26T21:49:41.179569+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:54.096536+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"470'8676","reported_seq":11556,"reported_epoch":470,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T21:49:43.236663+0000","last_change":"2024-03-26T21:49:30.159143+0000","last_active":"2024-03-26T21:49:43.236663+0000","last_peered":"2024-03-26T21:49:43.236663+0000","last_clean":"2024-03-26T21:48:57.973612+0000","last_became_active":"2024-03-26T21:49:00.120902+0000","last_became_peered":"2024-03-26T21:49:00.120902+0000","last_unstale":"2024-03-26T21:49:43.236663+0000","last_undegraded":"2024-03-26T21:49:43.236663+0000","last_fullsized":"2024-03-26T21:49:43.236663+0000","mapping_epoch":462,"log_start":"450'4356","ondisk_log_start":"450'4356","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422182912,"num_objects":6442,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5117,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1393,"num_bytes_recovered":91291648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":6442}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.184313+0000","last_change":"2024-03-26T21:49:29.979212+0000","last_active":"2024-03-26T21:49:41.184313+0000","last_peered":"2024-03-26T21:49:41.184313+0000","last_clean":"2024-03-26T21:49:41.184313+0000","last_became_active":"2024-03-26T21:49:29.978839+0000","last_became_peered":"2024-03-26T21:49:29.978839+0000","last_unstale":"2024-03-26T21:49:41.184313+0000","last_undegraded":"2024-03-26T21:49:41.184313+0000","last_fullsized":"2024-03-26T21:49:41.184313+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.183839+0000","last_change":"2024-03-26T21:48:35.997876+0000","last_active":"2024-03-26T21:49:41.183839+0000","last_peered":"2024-03-26T21:49:41.183839+0000","last_clean":"2024-03-26T21:49:41.183839+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:49:41.183839+0000","last_undegraded":"2024-03-26T21:49:41.183839+0000","last_fullsized":"2024-03-26T21:49:41.183839+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:48:35.997722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.997722+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:48.429570+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":913,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180937+0000","last_change":"2024-03-26T21:48:58.981784+0000","last_active":"2024-03-26T21:49:41.180937+0000","last_peered":"2024-03-26T21:49:41.180937+0000","last_clean":"2024-03-26T21:49:41.180937+0000","last_became_active":"2024-03-26T21:48:58.981168+0000","last_became_peered":"2024-03-26T21:48:58.981168+0000","last_unstale":"2024-03-26T21:49:41.180937+0000","last_undegraded":"2024-03-26T21:49:41.180937+0000","last_fullsized":"2024-03-26T21:49:41.180937+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":893,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176756+0000","last_change":"2024-03-26T21:48:58.983664+0000","last_active":"2024-03-26T21:49:41.176756+0000","last_peered":"2024-03-26T21:49:41.176756+0000","last_clean":"2024-03-26T21:49:41.176756+0000","last_became_active":"2024-03-26T21:48:58.983312+0000","last_became_peered":"2024-03-26T21:48:58.983312+0000","last_unstale":"2024-03-26T21:49:41.176756+0000","last_undegraded":"2024-03-26T21:49:41.176756+0000","last_fullsized":"2024-03-26T21:49:41.176756+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.809302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:04.683979+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"470'8526","reported_seq":8610,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.104199+0000","last_change":"2024-03-26T21:47:39.047854+0000","last_active":"2024-03-26T21:49:43.104199+0000","last_peered":"2024-03-26T21:49:43.104199+0000","last_clean":"2024-03-26T21:49:43.104199+0000","last_became_active":"2024-03-26T21:47:39.047314+0000","last_became_peered":"2024-03-26T21:47:39.047314+0000","last_unstale":"2024-03-26T21:49:43.104199+0000","last_undegraded":"2024-03-26T21:49:43.104199+0000","last_fullsized":"2024-03-26T21:49:43.104199+0000","mapping_epoch":433,"log_start":"449'3918","ondisk_log_start":"449'3918","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412745728,"num_objects":6298,"num_object_clones":0,"num_object_copies":18894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":888,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174889+0000","last_change":"2024-03-26T21:49:30.726913+0000","last_active":"2024-03-26T21:49:41.174889+0000","last_peered":"2024-03-26T21:49:41.174889+0000","last_clean":"2024-03-26T21:49:41.174889+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:49:41.174889+0000","last_undegraded":"2024-03-26T21:49:41.174889+0000","last_fullsized":"2024-03-26T21:49:41.174889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:30.726780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:30.726780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.940297+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":972,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180103+0000","last_change":"2024-03-26T21:48:59.007492+0000","last_active":"2024-03-26T21:49:41.180103+0000","last_peered":"2024-03-26T21:49:41.180103+0000","last_clean":"2024-03-26T21:49:41.180103+0000","last_became_active":"2024-03-26T21:48:58.986240+0000","last_became_peered":"2024-03-26T21:48:58.986240+0000","last_unstale":"2024-03-26T21:49:41.180103+0000","last_undegraded":"2024-03-26T21:49:41.180103+0000","last_fullsized":"2024-03-26T21:49:41.180103+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2004,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.598793+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:49:41.598793+0000","last_peered":"2024-03-26T21:49:41.598793+0000","last_clean":"2024-03-26T21:49:41.598793+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:49:41.598793+0000","last_undegraded":"2024-03-26T21:49:41.598793+0000","last_fullsized":"2024-03-26T21:49:41.598793+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1147,"num_read_kb":1142,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":916,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179361+0000","last_change":"2024-03-26T21:48:41.039131+0000","last_active":"2024-03-26T21:49:41.179361+0000","last_peered":"2024-03-26T21:49:41.179361+0000","last_clean":"2024-03-26T21:49:41.179361+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:49:41.179361+0000","last_undegraded":"2024-03-26T21:49:41.179361+0000","last_fullsized":"2024-03-26T21:49:41.179361+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.035559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.035559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:50.713534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":899,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177652+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:49:41.177652+0000","last_peered":"2024-03-26T21:49:41.177652+0000","last_clean":"2024-03-26T21:49:41.177652+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:49:41.177652+0000","last_undegraded":"2024-03-26T21:49:41.177652+0000","last_fullsized":"2024-03-26T21:49:41.177652+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"470'8665","reported_seq":8749,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.141598+0000","last_change":"2024-03-26T21:47:39.047378+0000","last_active":"2024-03-26T21:49:43.141598+0000","last_peered":"2024-03-26T21:49:43.141598+0000","last_clean":"2024-03-26T21:49:43.141598+0000","last_became_active":"2024-03-26T21:47:39.047073+0000","last_became_peered":"2024-03-26T21:47:39.047073+0000","last_unstale":"2024-03-26T21:49:43.141598+0000","last_undegraded":"2024-03-26T21:49:43.141598+0000","last_fullsized":"2024-03-26T21:49:43.141598+0000","mapping_epoch":433,"log_start":"449'4074","ondisk_log_start":"449'4074","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420151296,"num_objects":6411,"num_object_clones":0,"num_object_copies":19233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8665,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":825,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168966+0000","last_change":"2024-03-26T21:48:58.992875+0000","last_active":"2024-03-26T21:49:41.168966+0000","last_peered":"2024-03-26T21:49:41.168966+0000","last_clean":"2024-03-26T21:49:41.168966+0000","last_became_active":"2024-03-26T21:48:58.992117+0000","last_became_peered":"2024-03-26T21:48:58.992117+0000","last_unstale":"2024-03-26T21:49:41.168966+0000","last_undegraded":"2024-03-26T21:49:41.168966+0000","last_fullsized":"2024-03-26T21:49:41.168966+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"468'55","reported_seq":1263,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.182285+0000","last_change":"2024-03-26T21:49:29.985516+0000","last_active":"2024-03-26T21:49:41.182285+0000","last_peered":"2024-03-26T21:49:41.182285+0000","last_clean":"2024-03-26T21:49:41.182285+0000","last_became_active":"2024-03-26T21:49:29.985117+0000","last_became_peered":"2024-03-26T21:49:29.985117+0000","last_unstale":"2024-03-26T21:49:41.182285+0000","last_undegraded":"2024-03-26T21:49:41.182285+0000","last_fullsized":"2024-03-26T21:49:41.182285+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.195852+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:49:41.195852+0000","last_peered":"2024-03-26T21:49:41.195852+0000","last_clean":"2024-03-26T21:49:41.195852+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:49:41.195852+0000","last_undegraded":"2024-03-26T21:49:41.195852+0000","last_fullsized":"2024-03-26T21:49:41.195852+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1087,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168523+0000","last_change":"2024-03-26T21:48:27.821566+0000","last_active":"2024-03-26T21:49:41.168523+0000","last_peered":"2024-03-26T21:49:41.168523+0000","last_clean":"2024-03-26T21:49:41.168523+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:49:41.168523+0000","last_undegraded":"2024-03-26T21:49:41.168523+0000","last_fullsized":"2024-03-26T21:49:41.168523+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:48:27.821430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.821430+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:46.416532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":909,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174636+0000","last_change":"2024-03-26T21:49:29.780022+0000","last_active":"2024-03-26T21:49:41.174636+0000","last_peered":"2024-03-26T21:49:41.174636+0000","last_clean":"2024-03-26T21:49:41.174636+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:49:41.174636+0000","last_undegraded":"2024-03-26T21:49:41.174636+0000","last_fullsized":"2024-03-26T21:49:41.174636+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:29.779883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:29.779883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:46.086766+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"470'8728","reported_seq":8812,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.203659+0000","last_change":"2024-03-26T21:47:39.047146+0000","last_active":"2024-03-26T21:49:43.203659+0000","last_peered":"2024-03-26T21:49:43.203659+0000","last_clean":"2024-03-26T21:49:43.203659+0000","last_became_active":"2024-03-26T21:47:39.046837+0000","last_became_peered":"2024-03-26T21:47:39.046837+0000","last_unstale":"2024-03-26T21:49:43.203659+0000","last_undegraded":"2024-03-26T21:49:43.203659+0000","last_fullsized":"2024-03-26T21:49:43.203659+0000","mapping_epoch":433,"log_start":"449'4134","ondisk_log_start":"449'4134","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417595392,"num_objects":6372,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8728,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":504,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173892+0000","last_change":"2024-03-26T21:49:19.802374+0000","last_active":"2024-03-26T21:49:41.173892+0000","last_peered":"2024-03-26T21:49:41.173892+0000","last_clean":"2024-03-26T21:49:41.173892+0000","last_became_active":"2024-03-26T21:48:58.985930+0000","last_became_peered":"2024-03-26T21:48:58.985930+0000","last_unstale":"2024-03-26T21:49:41.173892+0000","last_undegraded":"2024-03-26T21:49:41.173892+0000","last_fullsized":"2024-03-26T21:49:41.173892+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:19.802252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:49:19.802252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:21.943907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":946,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.186631+0000","last_change":"2024-03-26T21:48:35.767934+0000","last_active":"2024-03-26T21:49:41.186631+0000","last_peered":"2024-03-26T21:49:41.186631+0000","last_clean":"2024-03-26T21:49:41.186631+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:49:41.186631+0000","last_undegraded":"2024-03-26T21:49:41.186631+0000","last_fullsized":"2024-03-26T21:49:41.186631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:48:35.767034+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.767034+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:00.271391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":714,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174449+0000","last_change":"2024-03-26T21:48:58.978047+0000","last_active":"2024-03-26T21:49:41.174449+0000","last_peered":"2024-03-26T21:49:41.174449+0000","last_clean":"2024-03-26T21:49:41.174449+0000","last_became_active":"2024-03-26T21:48:58.976535+0000","last_became_peered":"2024-03-26T21:48:58.976535+0000","last_unstale":"2024-03-26T21:49:41.174449+0000","last_undegraded":"2024-03-26T21:49:41.174449+0000","last_fullsized":"2024-03-26T21:49:41.174449+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":913,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177503+0000","last_change":"2024-03-26T21:48:58.982624+0000","last_active":"2024-03-26T21:49:41.177503+0000","last_peered":"2024-03-26T21:49:41.177503+0000","last_clean":"2024-03-26T21:49:41.177503+0000","last_became_active":"2024-03-26T21:48:58.982052+0000","last_became_peered":"2024-03-26T21:48:58.982052+0000","last_unstale":"2024-03-26T21:49:41.177503+0000","last_undegraded":"2024-03-26T21:49:41.177503+0000","last_fullsized":"2024-03-26T21:49:41.177503+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":807,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169605+0000","last_change":"2024-03-26T21:48:31.817091+0000","last_active":"2024-03-26T21:49:41.169605+0000","last_peered":"2024-03-26T21:49:41.169605+0000","last_clean":"2024-03-26T21:49:41.169605+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:49:41.169605+0000","last_undegraded":"2024-03-26T21:49:41.169605+0000","last_fullsized":"2024-03-26T21:49:41.169605+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:31.817002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.817002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:01.719710+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"470'8454","reported_seq":8538,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.240748+0000","last_change":"2024-03-26T21:47:39.051677+0000","last_active":"2024-03-26T21:49:43.240748+0000","last_peered":"2024-03-26T21:49:43.240748+0000","last_clean":"2024-03-26T21:49:43.240748+0000","last_became_active":"2024-03-26T21:47:39.051279+0000","last_became_peered":"2024-03-26T21:47:39.051279+0000","last_unstale":"2024-03-26T21:49:43.240748+0000","last_undegraded":"2024-03-26T21:49:43.240748+0000","last_fullsized":"2024-03-26T21:49:43.240748+0000","mapping_epoch":433,"log_start":"449'3904","ondisk_log_start":"449'3904","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410517504,"num_objects":6264,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":470976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":896,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180077+0000","last_change":"2024-03-26T21:48:32.000228+0000","last_active":"2024-03-26T21:49:41.180077+0000","last_peered":"2024-03-26T21:49:41.180077+0000","last_clean":"2024-03-26T21:49:41.180077+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:49:41.180077+0000","last_undegraded":"2024-03-26T21:49:41.180077+0000","last_fullsized":"2024-03-26T21:49:41.180077+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.000079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"462'56","reported_seq":1262,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173880+0000","last_change":"2024-03-26T21:48:59.005174+0000","last_active":"2024-03-26T21:49:41.173880+0000","last_peered":"2024-03-26T21:49:41.173880+0000","last_clean":"2024-03-26T21:49:41.173880+0000","last_became_active":"2024-03-26T21:48:58.981458+0000","last_became_peered":"2024-03-26T21:48:58.981458+0000","last_unstale":"2024-03-26T21:49:41.173880+0000","last_undegraded":"2024-03-26T21:49:41.173880+0000","last_fullsized":"2024-03-26T21:49:41.173880+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'43","last_scrub_stamp":"2024-03-26T21:48:14.657283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.657283+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.892043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178318+0000","last_change":"2024-03-26T21:48:58.976086+0000","last_active":"2024-03-26T21:49:41.178318+0000","last_peered":"2024-03-26T21:49:41.178318+0000","last_clean":"2024-03-26T21:49:41.178318+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:49:41.178318+0000","last_undegraded":"2024-03-26T21:49:41.178318+0000","last_fullsized":"2024-03-26T21:49:41.178318+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":907,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173540+0000","last_change":"2024-03-26T21:48:43.783835+0000","last_active":"2024-03-26T21:49:41.173540+0000","last_peered":"2024-03-26T21:49:41.173540+0000","last_clean":"2024-03-26T21:49:41.173540+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:49:41.173540+0000","last_undegraded":"2024-03-26T21:49:41.173540+0000","last_fullsized":"2024-03-26T21:49:41.173540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:48:43.783585+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:07.474939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174546+0000","last_change":"2024-03-26T21:48:29.771118+0000","last_active":"2024-03-26T21:49:41.174546+0000","last_peered":"2024-03-26T21:49:41.174546+0000","last_clean":"2024-03-26T21:49:41.174546+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:49:41.174546+0000","last_undegraded":"2024-03-26T21:49:41.174546+0000","last_fullsized":"2024-03-26T21:49:41.174546+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.770950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.770950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"470'8679","reported_seq":18917,"reported_epoch":470,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T21:49:43.486322+0000","last_change":"2024-03-26T21:49:00.197127+0000","last_active":"2024-03-26T21:49:43.486322+0000","last_peered":"2024-03-26T21:49:43.486322+0000","last_clean":"2024-03-26T21:48:57.958781+0000","last_became_active":"2024-03-26T21:49:00.175941+0000","last_became_peered":"2024-03-26T21:49:00.175941+0000","last_unstale":"2024-03-26T21:49:43.486322+0000","last_undegraded":"2024-03-26T21:49:43.486322+0000","last_fullsized":"2024-03-26T21:49:43.486322+0000","mapping_epoch":462,"log_start":"449'4024","ondisk_log_start":"449'4024","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4655,"log_dups_size":0,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413073408,"num_objects":6303,"num_object_clones":0,"num_object_copies":18909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1663,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8679,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5074,"num_bytes_recovered":332529664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,2],"avail_no_missing":["7","2","6"],"object_location_counts":[{"shards":"2,6,7","objects":6303}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":937,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.183976+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:49:41.183976+0000","last_peered":"2024-03-26T21:49:41.183976+0000","last_clean":"2024-03-26T21:49:41.183976+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:49:41.183976+0000","last_undegraded":"2024-03-26T21:49:41.183976+0000","last_fullsized":"2024-03-26T21:49:41.183976+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":916,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174220+0000","last_change":"2024-03-26T21:49:31.653285+0000","last_active":"2024-03-26T21:49:41.174220+0000","last_peered":"2024-03-26T21:49:41.174220+0000","last_clean":"2024-03-26T21:49:41.174220+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:49:41.174220+0000","last_undegraded":"2024-03-26T21:49:41.174220+0000","last_fullsized":"2024-03-26T21:49:41.174220+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:31.653167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:49:31.653167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:59.137490+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":712,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177652+0000","last_change":"2024-03-26T21:48:58.968424+0000","last_active":"2024-03-26T21:49:41.177652+0000","last_peered":"2024-03-26T21:49:41.177652+0000","last_clean":"2024-03-26T21:49:41.177652+0000","last_became_active":"2024-03-26T21:48:58.968010+0000","last_became_peered":"2024-03-26T21:48:58.968010+0000","last_unstale":"2024-03-26T21:49:41.177652+0000","last_undegraded":"2024-03-26T21:49:41.177652+0000","last_fullsized":"2024-03-26T21:49:41.177652+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":877,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179579+0000","last_change":"2024-03-26T21:48:58.983049+0000","last_active":"2024-03-26T21:49:41.179579+0000","last_peered":"2024-03-26T21:49:41.179579+0000","last_clean":"2024-03-26T21:49:41.179579+0000","last_became_active":"2024-03-26T21:48:58.982744+0000","last_became_peered":"2024-03-26T21:48:58.982744+0000","last_unstale":"2024-03-26T21:49:41.179579+0000","last_undegraded":"2024-03-26T21:49:41.179579+0000","last_fullsized":"2024-03-26T21:49:41.179579+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"470'8355","reported_seq":8440,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:43.490904+0000","last_change":"2024-03-26T21:47:39.046775+0000","last_active":"2024-03-26T21:49:43.490904+0000","last_peered":"2024-03-26T21:49:43.490904+0000","last_clean":"2024-03-26T21:49:43.490904+0000","last_became_active":"2024-03-26T21:47:39.046369+0000","last_became_peered":"2024-03-26T21:47:39.046369+0000","last_unstale":"2024-03-26T21:49:43.490904+0000","last_undegraded":"2024-03-26T21:49:43.490904+0000","last_fullsized":"2024-03-26T21:49:43.490904+0000","mapping_epoch":433,"log_start":"449'3657","ondisk_log_start":"449'3657","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402325504,"num_objects":6139,"num_object_clones":0,"num_object_copies":18417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":977,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177348+0000","last_change":"2024-03-26T21:49:29.981307+0000","last_active":"2024-03-26T21:49:41.177348+0000","last_peered":"2024-03-26T21:49:41.177348+0000","last_clean":"2024-03-26T21:49:41.177348+0000","last_became_active":"2024-03-26T21:49:29.980914+0000","last_became_peered":"2024-03-26T21:49:29.980914+0000","last_unstale":"2024-03-26T21:49:41.177348+0000","last_undegraded":"2024-03-26T21:49:41.177348+0000","last_fullsized":"2024-03-26T21:49:41.177348+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":841,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169620+0000","last_change":"2024-03-26T21:48:44.612681+0000","last_active":"2024-03-26T21:49:41.169620+0000","last_peered":"2024-03-26T21:49:41.169620+0000","last_clean":"2024-03-26T21:49:41.169620+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:49:41.169620+0000","last_undegraded":"2024-03-26T21:49:41.169620+0000","last_fullsized":"2024-03-26T21:49:41.169620+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.612470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.145480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":28,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.183383+0000","last_change":"2024-03-26T21:48:58.986135+0000","last_active":"2024-03-26T21:49:41.183383+0000","last_peered":"2024-03-26T21:49:41.183383+0000","last_clean":"2024-03-26T21:49:41.183383+0000","last_became_active":"2024-03-26T21:48:58.985788+0000","last_became_peered":"2024-03-26T21:48:58.985788+0000","last_unstale":"2024-03-26T21:49:41.183383+0000","last_undegraded":"2024-03-26T21:49:41.183383+0000","last_fullsized":"2024-03-26T21:49:41.183383+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":812,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168206+0000","last_change":"2024-03-26T21:48:58.984676+0000","last_active":"2024-03-26T21:49:41.168206+0000","last_peered":"2024-03-26T21:49:41.168206+0000","last_clean":"2024-03-26T21:49:41.168206+0000","last_became_active":"2024-03-26T21:48:58.983980+0000","last_became_peered":"2024-03-26T21:48:58.983980+0000","last_unstale":"2024-03-26T21:49:41.168206+0000","last_undegraded":"2024-03-26T21:49:41.168206+0000","last_fullsized":"2024-03-26T21:49:41.168206+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"470'8546","reported_seq":22147,"reported_epoch":470,"state":"active+clean+remapped","last_fresh":"2024-03-26T21:49:44.044543+0000","last_change":"2024-03-26T21:49:43.329891+0000","last_active":"2024-03-26T21:49:44.044543+0000","last_peered":"2024-03-26T21:49:44.044543+0000","last_clean":"2024-03-26T21:49:44.044543+0000","last_became_active":"2024-03-26T21:49:00.083242+0000","last_became_peered":"2024-03-26T21:49:00.083242+0000","last_unstale":"2024-03-26T21:49:44.044543+0000","last_undegraded":"2024-03-26T21:49:44.044543+0000","last_fullsized":"2024-03-26T21:49:44.044543+0000","mapping_epoch":462,"log_start":"449'3711","ondisk_log_start":"449'3711","created":433,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402653184,"num_objects":6144,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6755,"num_bytes_recovered":442695680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,2],"avail_no_missing":["0","1","2","4"],"object_location_counts":[{"shards":"0,1,2","objects":6144}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":28,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.181844+0000","last_change":"2024-03-26T21:48:58.977207+0000","last_active":"2024-03-26T21:49:41.181844+0000","last_peered":"2024-03-26T21:49:41.181844+0000","last_clean":"2024-03-26T21:49:41.181844+0000","last_became_active":"2024-03-26T21:48:58.976826+0000","last_became_peered":"2024-03-26T21:48:58.976826+0000","last_unstale":"2024-03-26T21:49:41.181844+0000","last_undegraded":"2024-03-26T21:49:41.181844+0000","last_fullsized":"2024-03-26T21:49:41.181844+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":649,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177953+0000","last_change":"2024-03-26T21:48:36.706426+0000","last_active":"2024-03-26T21:49:41.177953+0000","last_peered":"2024-03-26T21:49:41.177953+0000","last_clean":"2024-03-26T21:49:41.177953+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:49:41.177953+0000","last_undegraded":"2024-03-26T21:49:41.177953+0000","last_fullsized":"2024-03-26T21:49:41.177953+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:48:36.706278+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":871,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169456+0000","last_change":"2024-03-26T21:48:35.727970+0000","last_active":"2024-03-26T21:49:41.169456+0000","last_peered":"2024-03-26T21:49:41.169456+0000","last_clean":"2024-03-26T21:49:41.169456+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:49:41.169456+0000","last_undegraded":"2024-03-26T21:49:41.169456+0000","last_fullsized":"2024-03-26T21:49:41.169456+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.727841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.817821+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177078+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:49:41.177078+0000","last_peered":"2024-03-26T21:49:41.177078+0000","last_clean":"2024-03-26T21:49:41.177078+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:49:41.177078+0000","last_undegraded":"2024-03-26T21:49:41.177078+0000","last_fullsized":"2024-03-26T21:49:41.177078+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"470'8554","reported_seq":8719,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:42.134312+0000","last_change":"2024-03-26T21:49:25.777937+0000","last_active":"2024-03-26T21:49:42.134312+0000","last_peered":"2024-03-26T21:49:42.134312+0000","last_clean":"2024-03-26T21:49:42.134312+0000","last_became_active":"2024-03-26T21:47:39.051349+0000","last_became_peered":"2024-03-26T21:47:39.051349+0000","last_unstale":"2024-03-26T21:49:42.134312+0000","last_undegraded":"2024-03-26T21:49:42.134312+0000","last_fullsized":"2024-03-26T21:49:42.134312+0000","mapping_epoch":433,"log_start":"446'3237","ondisk_log_start":"446'3237","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'7654","last_scrub_stamp":"2024-03-26T21:49:25.777551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:49:25.777551+0000","objects_scrubbed":7294,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:47.024151+0000","scrub_duration":2010,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105024,"num_objects":6334,"num_object_clones":0,"num_object_copies":19002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1080,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168338+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:49:41.168338+0000","last_peered":"2024-03-26T21:49:41.168338+0000","last_clean":"2024-03-26T21:49:41.168338+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:49:41.168338+0000","last_undegraded":"2024-03-26T21:49:41.168338+0000","last_fullsized":"2024-03-26T21:49:41.168338+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":969,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173320+0000","last_change":"2024-03-26T21:49:38.645618+0000","last_active":"2024-03-26T21:49:41.173320+0000","last_peered":"2024-03-26T21:49:41.173320+0000","last_clean":"2024-03-26T21:49:41.173320+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:49:41.173320+0000","last_undegraded":"2024-03-26T21:49:41.173320+0000","last_fullsized":"2024-03-26T21:49:41.173320+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:49:38.645456+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.832210+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":880,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169692+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:49:41.169692+0000","last_peered":"2024-03-26T21:49:41.169692+0000","last_clean":"2024-03-26T21:49:41.169692+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:49:41.169692+0000","last_undegraded":"2024-03-26T21:49:41.169692+0000","last_fullsized":"2024-03-26T21:49:41.169692+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":884,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173782+0000","last_change":"2024-03-26T21:49:35.727593+0000","last_active":"2024-03-26T21:49:41.173782+0000","last_peered":"2024-03-26T21:49:41.173782+0000","last_clean":"2024-03-26T21:49:41.173782+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:49:41.173782+0000","last_undegraded":"2024-03-26T21:49:41.173782+0000","last_fullsized":"2024-03-26T21:49:41.173782+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:35.727500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:35.727500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:52.939494+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":264,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176872+0000","last_change":"2024-03-26T21:48:59.002050+0000","last_active":"2024-03-26T21:49:41.176872+0000","last_peered":"2024-03-26T21:49:41.176872+0000","last_clean":"2024-03-26T21:49:41.176872+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:49:41.176872+0000","last_undegraded":"2024-03-26T21:49:41.176872+0000","last_fullsized":"2024-03-26T21:49:41.176872+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.181933+0000","last_change":"2024-03-26T21:49:29.977391+0000","last_active":"2024-03-26T21:49:41.181933+0000","last_peered":"2024-03-26T21:49:41.181933+0000","last_clean":"2024-03-26T21:49:41.181933+0000","last_became_active":"2024-03-26T21:49:29.976836+0000","last_became_peered":"2024-03-26T21:49:29.976836+0000","last_unstale":"2024-03-26T21:49:41.181933+0000","last_undegraded":"2024-03-26T21:49:41.181933+0000","last_fullsized":"2024-03-26T21:49:41.181933+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.291965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:59.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":899,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.704967+0000","last_change":"2024-03-26T21:49:41.705007+0000","last_active":"2024-03-26T21:49:41.704967+0000","last_peered":"2024-03-26T21:49:41.704967+0000","last_clean":"2024-03-26T21:49:41.704967+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:49:41.704967+0000","last_undegraded":"2024-03-26T21:49:41.704967+0000","last_fullsized":"2024-03-26T21:49:41.704967+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:41.704871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:41.704871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:49.588465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":816,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169264+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:49:41.169264+0000","last_peered":"2024-03-26T21:49:41.169264+0000","last_clean":"2024-03-26T21:49:41.169264+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:49:41.169264+0000","last_undegraded":"2024-03-26T21:49:41.169264+0000","last_fullsized":"2024-03-26T21:49:41.169264+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180656+0000","last_change":"2024-03-26T21:48:44.101005+0000","last_active":"2024-03-26T21:49:41.180656+0000","last_peered":"2024-03-26T21:49:41.180656+0000","last_clean":"2024-03-26T21:49:41.180656+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:49:41.180656+0000","last_undegraded":"2024-03-26T21:49:41.180656+0000","last_fullsized":"2024-03-26T21:49:41.180656+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:48:44.100867+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.100867+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:07.849510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":853,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168532+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:49:41.168532+0000","last_peered":"2024-03-26T21:49:41.168532+0000","last_clean":"2024-03-26T21:49:41.168532+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:49:41.168532+0000","last_undegraded":"2024-03-26T21:49:41.168532+0000","last_fullsized":"2024-03-26T21:49:41.168532+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177173+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:49:41.177173+0000","last_peered":"2024-03-26T21:49:41.177173+0000","last_clean":"2024-03-26T21:49:41.177173+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:49:41.177173+0000","last_undegraded":"2024-03-26T21:49:41.177173+0000","last_fullsized":"2024-03-26T21:49:41.177173+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":513,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177680+0000","last_change":"2024-03-26T21:48:58.984295+0000","last_active":"2024-03-26T21:49:41.177680+0000","last_peered":"2024-03-26T21:49:41.177680+0000","last_clean":"2024-03-26T21:49:41.177680+0000","last_became_active":"2024-03-26T21:48:58.983139+0000","last_became_peered":"2024-03-26T21:48:58.983139+0000","last_unstale":"2024-03-26T21:49:41.177680+0000","last_undegraded":"2024-03-26T21:49:41.177680+0000","last_fullsized":"2024-03-26T21:49:41.177680+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":960,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.175247+0000","last_change":"2024-03-26T21:48:20.776635+0000","last_active":"2024-03-26T21:49:41.175247+0000","last_peered":"2024-03-26T21:49:41.175247+0000","last_clean":"2024-03-26T21:49:41.175247+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:49:41.175247+0000","last_undegraded":"2024-03-26T21:49:41.175247+0000","last_fullsized":"2024-03-26T21:49:41.175247+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:20.776494+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.776494+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:44.363911+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":893,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.184286+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:49:41.184286+0000","last_peered":"2024-03-26T21:49:41.184286+0000","last_clean":"2024-03-26T21:49:41.184286+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:49:41.184286+0000","last_undegraded":"2024-03-26T21:49:41.184286+0000","last_fullsized":"2024-03-26T21:49:41.184286+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"462'14","reported_seq":307,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176519+0000","last_change":"2024-03-26T21:48:58.993774+0000","last_active":"2024-03-26T21:49:41.176519+0000","last_peered":"2024-03-26T21:49:41.176519+0000","last_clean":"2024-03-26T21:49:41.176519+0000","last_became_active":"2024-03-26T21:48:58.989325+0000","last_became_peered":"2024-03-26T21:48:58.989325+0000","last_unstale":"2024-03-26T21:49:41.176519+0000","last_undegraded":"2024-03-26T21:49:41.176519+0000","last_fullsized":"2024-03-26T21:49:41.176519+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13","last_scrub_stamp":"2024-03-26T21:48:33.859117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.859117+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:58.896284+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":583,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180764+0000","last_change":"2024-03-26T21:48:58.970848+0000","last_active":"2024-03-26T21:49:41.180764+0000","last_peered":"2024-03-26T21:49:41.180764+0000","last_clean":"2024-03-26T21:49:41.180764+0000","last_became_active":"2024-03-26T21:48:58.970509+0000","last_became_peered":"2024-03-26T21:48:58.970509+0000","last_unstale":"2024-03-26T21:49:41.180764+0000","last_undegraded":"2024-03-26T21:49:41.180764+0000","last_fullsized":"2024-03-26T21:49:41.180764+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":260,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176356+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:49:41.176356+0000","last_peered":"2024-03-26T21:49:41.176356+0000","last_clean":"2024-03-26T21:49:41.176356+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:49:41.176356+0000","last_undegraded":"2024-03-26T21:49:41.176356+0000","last_fullsized":"2024-03-26T21:49:41.176356+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":832,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169012+0000","last_change":"2024-03-26T21:48:58.987967+0000","last_active":"2024-03-26T21:49:41.169012+0000","last_peered":"2024-03-26T21:49:41.169012+0000","last_clean":"2024-03-26T21:49:41.169012+0000","last_became_active":"2024-03-26T21:48:58.987113+0000","last_became_peered":"2024-03-26T21:48:58.987113+0000","last_unstale":"2024-03-26T21:49:41.169012+0000","last_undegraded":"2024-03-26T21:49:41.169012+0000","last_fullsized":"2024-03-26T21:49:41.169012+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1142,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174668+0000","last_change":"2024-03-26T21:48:41.813232+0000","last_active":"2024-03-26T21:49:41.174668+0000","last_peered":"2024-03-26T21:49:41.174668+0000","last_clean":"2024-03-26T21:49:41.174668+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:49:41.174668+0000","last_undegraded":"2024-03-26T21:49:41.174668+0000","last_fullsized":"2024-03-26T21:49:41.174668+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:48:41.813092+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.813092+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.588630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":889,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179985+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:49:41.179985+0000","last_peered":"2024-03-26T21:49:41.179985+0000","last_clean":"2024-03-26T21:49:41.179985+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:49:41.179985+0000","last_undegraded":"2024-03-26T21:49:41.179985+0000","last_fullsized":"2024-03-26T21:49:41.179985+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":619,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178540+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:49:41.178540+0000","last_peered":"2024-03-26T21:49:41.178540+0000","last_clean":"2024-03-26T21:49:41.178540+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:49:41.178540+0000","last_undegraded":"2024-03-26T21:49:41.178540+0000","last_fullsized":"2024-03-26T21:49:41.178540+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":587,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179483+0000","last_change":"2024-03-26T21:48:39.735268+0000","last_active":"2024-03-26T21:49:41.179483+0000","last_peered":"2024-03-26T21:49:41.179483+0000","last_clean":"2024-03-26T21:49:41.179483+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:49:41.179483+0000","last_undegraded":"2024-03-26T21:49:41.179483+0000","last_fullsized":"2024-03-26T21:49:41.179483+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:48:39.735146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.746887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":914,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177458+0000","last_change":"2024-03-26T21:48:24.700288+0000","last_active":"2024-03-26T21:49:41.177458+0000","last_peered":"2024-03-26T21:49:41.177458+0000","last_clean":"2024-03-26T21:49:41.177458+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:49:41.177458+0000","last_undegraded":"2024-03-26T21:49:41.177458+0000","last_fullsized":"2024-03-26T21:49:41.177458+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:48:24.700160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:41.327860+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174602+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:49:41.174602+0000","last_peered":"2024-03-26T21:49:41.174602+0000","last_clean":"2024-03-26T21:49:41.174602+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:49:41.174602+0000","last_undegraded":"2024-03-26T21:49:41.174602+0000","last_fullsized":"2024-03-26T21:49:41.174602+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":948,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177654+0000","last_change":"2024-03-26T21:48:59.005271+0000","last_active":"2024-03-26T21:49:41.177654+0000","last_peered":"2024-03-26T21:49:41.177654+0000","last_clean":"2024-03-26T21:49:41.177654+0000","last_became_active":"2024-03-26T21:48:58.979497+0000","last_became_peered":"2024-03-26T21:48:58.979497+0000","last_unstale":"2024-03-26T21:49:41.177654+0000","last_undegraded":"2024-03-26T21:49:41.177654+0000","last_fullsized":"2024-03-26T21:49:41.177654+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:30.679017+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.679017+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:52.862815+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":831,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.169361+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:49:41.169361+0000","last_peered":"2024-03-26T21:49:41.169361+0000","last_clean":"2024-03-26T21:49:41.169361+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:49:41.169361+0000","last_undegraded":"2024-03-26T21:49:41.169361+0000","last_fullsized":"2024-03-26T21:49:41.169361+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176914+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:49:41.176914+0000","last_peered":"2024-03-26T21:49:41.176914+0000","last_clean":"2024-03-26T21:49:41.176914+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:49:41.176914+0000","last_undegraded":"2024-03-26T21:49:41.176914+0000","last_fullsized":"2024-03-26T21:49:41.176914+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":870,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179526+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:49:41.179526+0000","last_peered":"2024-03-26T21:49:41.179526+0000","last_clean":"2024-03-26T21:49:41.179526+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:49:41.179526+0000","last_undegraded":"2024-03-26T21:49:41.179526+0000","last_fullsized":"2024-03-26T21:49:41.179526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":265,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178080+0000","last_change":"2024-03-26T21:48:38.975837+0000","last_active":"2024-03-26T21:49:41.178080+0000","last_peered":"2024-03-26T21:49:41.178080+0000","last_clean":"2024-03-26T21:49:41.178080+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:49:41.178080+0000","last_undegraded":"2024-03-26T21:49:41.178080+0000","last_fullsized":"2024-03-26T21:49:41.178080+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:38.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:38.975710+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:05.250119+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":896,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173744+0000","last_change":"2024-03-26T21:48:12.733553+0000","last_active":"2024-03-26T21:49:41.173744+0000","last_peered":"2024-03-26T21:49:41.173744+0000","last_clean":"2024-03-26T21:49:41.173744+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:49:41.173744+0000","last_undegraded":"2024-03-26T21:49:41.173744+0000","last_fullsized":"2024-03-26T21:49:41.173744+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.733428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.733428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":808,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168485+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:49:41.168485+0000","last_peered":"2024-03-26T21:49:41.168485+0000","last_clean":"2024-03-26T21:49:41.168485+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:49:41.168485+0000","last_undegraded":"2024-03-26T21:49:41.168485+0000","last_fullsized":"2024-03-26T21:49:41.168485+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":241,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176995+0000","last_change":"2024-03-26T21:48:37.104233+0000","last_active":"2024-03-26T21:49:41.176995+0000","last_peered":"2024-03-26T21:49:41.176995+0000","last_clean":"2024-03-26T21:49:41.176995+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:49:41.176995+0000","last_undegraded":"2024-03-26T21:49:41.176995+0000","last_fullsized":"2024-03-26T21:49:41.176995+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.104108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.104108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:51.248220+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":253,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179196+0000","last_change":"2024-03-26T21:48:42.913651+0000","last_active":"2024-03-26T21:49:41.179196+0000","last_peered":"2024-03-26T21:49:41.179196+0000","last_clean":"2024-03-26T21:49:41.179196+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:49:41.179196+0000","last_undegraded":"2024-03-26T21:49:41.179196+0000","last_fullsized":"2024-03-26T21:49:41.179196+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:48:42.913438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:42.913438+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:46.025071+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":893,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.182606+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:49:41.182606+0000","last_peered":"2024-03-26T21:49:41.182606+0000","last_clean":"2024-03-26T21:49:41.182606+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:49:41.182606+0000","last_undegraded":"2024-03-26T21:49:41.182606+0000","last_fullsized":"2024-03-26T21:49:41.182606+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":505,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177680+0000","last_change":"2024-03-26T21:49:29.976926+0000","last_active":"2024-03-26T21:49:41.177680+0000","last_peered":"2024-03-26T21:49:41.177680+0000","last_clean":"2024-03-26T21:49:41.177680+0000","last_became_active":"2024-03-26T21:49:29.976388+0000","last_became_peered":"2024-03-26T21:49:29.976388+0000","last_unstale":"2024-03-26T21:49:41.177680+0000","last_undegraded":"2024-03-26T21:49:41.177680+0000","last_fullsized":"2024-03-26T21:49:41.177680+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":822,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168542+0000","last_change":"2024-03-26T21:48:58.980520+0000","last_active":"2024-03-26T21:49:41.168542+0000","last_peered":"2024-03-26T21:49:41.168542+0000","last_clean":"2024-03-26T21:49:41.168542+0000","last_became_active":"2024-03-26T21:48:58.980151+0000","last_became_peered":"2024-03-26T21:48:58.980151+0000","last_unstale":"2024-03-26T21:49:41.168542+0000","last_undegraded":"2024-03-26T21:49:41.168542+0000","last_fullsized":"2024-03-26T21:49:41.168542+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":936,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174790+0000","last_change":"2024-03-26T21:49:29.980586+0000","last_active":"2024-03-26T21:49:41.174790+0000","last_peered":"2024-03-26T21:49:41.174790+0000","last_clean":"2024-03-26T21:49:41.174790+0000","last_became_active":"2024-03-26T21:49:29.980233+0000","last_became_peered":"2024-03-26T21:49:29.980233+0000","last_unstale":"2024-03-26T21:49:41.174790+0000","last_undegraded":"2024-03-26T21:49:41.174790+0000","last_fullsized":"2024-03-26T21:49:41.174790+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:31.342662+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.342662+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178969+0000","last_change":"2024-03-26T21:48:26.761549+0000","last_active":"2024-03-26T21:49:41.178969+0000","last_peered":"2024-03-26T21:49:41.178969+0000","last_clean":"2024-03-26T21:49:41.178969+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:49:41.178969+0000","last_undegraded":"2024-03-26T21:49:41.178969+0000","last_fullsized":"2024-03-26T21:49:41.178969+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.760812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.760812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":890,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180801+0000","last_change":"2024-03-26T21:48:58.986355+0000","last_active":"2024-03-26T21:49:41.180801+0000","last_peered":"2024-03-26T21:49:41.180801+0000","last_clean":"2024-03-26T21:49:41.180801+0000","last_became_active":"2024-03-26T21:48:58.985913+0000","last_became_peered":"2024-03-26T21:48:58.985913+0000","last_unstale":"2024-03-26T21:49:41.180801+0000","last_undegraded":"2024-03-26T21:49:41.180801+0000","last_fullsized":"2024-03-26T21:49:41.180801+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:48:45.073486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:50.471690+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":906,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178579+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:49:41.178579+0000","last_peered":"2024-03-26T21:49:41.178579+0000","last_clean":"2024-03-26T21:49:41.178579+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:49:41.178579+0000","last_undegraded":"2024-03-26T21:49:41.178579+0000","last_fullsized":"2024-03-26T21:49:41.178579+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:47.137319+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":977,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180554+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:49:41.180554+0000","last_peered":"2024-03-26T21:49:41.180554+0000","last_clean":"2024-03-26T21:49:41.180554+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:49:41.180554+0000","last_undegraded":"2024-03-26T21:49:41.180554+0000","last_fullsized":"2024-03-26T21:49:41.180554+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":940,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176905+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:49:41.176905+0000","last_peered":"2024-03-26T21:49:41.176905+0000","last_clean":"2024-03-26T21:49:41.176905+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:49:41.176905+0000","last_undegraded":"2024-03-26T21:49:41.176905+0000","last_fullsized":"2024-03-26T21:49:41.176905+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"468'15","reported_seq":463,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179685+0000","last_change":"2024-03-26T21:49:29.976278+0000","last_active":"2024-03-26T21:49:41.179685+0000","last_peered":"2024-03-26T21:49:41.179685+0000","last_clean":"2024-03-26T21:49:41.179685+0000","last_became_active":"2024-03-26T21:49:29.975921+0000","last_became_peered":"2024-03-26T21:49:29.975921+0000","last_unstale":"2024-03-26T21:49:41.179685+0000","last_undegraded":"2024-03-26T21:49:41.179685+0000","last_fullsized":"2024-03-26T21:49:41.179685+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:48:32.327875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.327875+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:45.500786+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":822,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168798+0000","last_change":"2024-03-26T21:48:58.998136+0000","last_active":"2024-03-26T21:49:41.168798+0000","last_peered":"2024-03-26T21:49:41.168798+0000","last_clean":"2024-03-26T21:49:41.168798+0000","last_became_active":"2024-03-26T21:48:58.991425+0000","last_became_peered":"2024-03-26T21:48:58.991425+0000","last_unstale":"2024-03-26T21:49:41.168798+0000","last_undegraded":"2024-03-26T21:49:41.168798+0000","last_fullsized":"2024-03-26T21:49:41.168798+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":258,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174387+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:49:41.174387+0000","last_peered":"2024-03-26T21:49:41.174387+0000","last_clean":"2024-03-26T21:49:41.174387+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:49:41.174387+0000","last_undegraded":"2024-03-26T21:49:41.174387+0000","last_fullsized":"2024-03-26T21:49:41.174387+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":584,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178005+0000","last_change":"2024-03-26T21:48:58.973845+0000","last_active":"2024-03-26T21:49:41.178005+0000","last_peered":"2024-03-26T21:49:41.178005+0000","last_clean":"2024-03-26T21:49:41.178005+0000","last_became_active":"2024-03-26T21:48:58.973428+0000","last_became_peered":"2024-03-26T21:48:58.973428+0000","last_unstale":"2024-03-26T21:49:41.178005+0000","last_undegraded":"2024-03-26T21:49:41.178005+0000","last_fullsized":"2024-03-26T21:49:41.178005+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":608,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179165+0000","last_change":"2024-03-26T21:48:58.983537+0000","last_active":"2024-03-26T21:49:41.179165+0000","last_peered":"2024-03-26T21:49:41.179165+0000","last_clean":"2024-03-26T21:49:41.179165+0000","last_became_active":"2024-03-26T21:48:58.982850+0000","last_became_peered":"2024-03-26T21:48:58.982850+0000","last_unstale":"2024-03-26T21:49:41.179165+0000","last_undegraded":"2024-03-26T21:49:41.179165+0000","last_fullsized":"2024-03-26T21:49:41.179165+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":233,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177223+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:49:41.177223+0000","last_peered":"2024-03-26T21:49:41.177223+0000","last_clean":"2024-03-26T21:49:41.177223+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:49:41.177223+0000","last_undegraded":"2024-03-26T21:49:41.177223+0000","last_fullsized":"2024-03-26T21:49:41.177223+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1020,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.180833+0000","last_change":"2024-03-26T21:48:59.014048+0000","last_active":"2024-03-26T21:49:41.180833+0000","last_peered":"2024-03-26T21:49:41.180833+0000","last_clean":"2024-03-26T21:49:41.180833+0000","last_became_active":"2024-03-26T21:48:58.977341+0000","last_became_peered":"2024-03-26T21:48:58.977341+0000","last_unstale":"2024-03-26T21:49:41.180833+0000","last_undegraded":"2024-03-26T21:49:41.180833+0000","last_fullsized":"2024-03-26T21:49:41.180833+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":903,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.183754+0000","last_change":"2024-03-26T21:48:40.024866+0000","last_active":"2024-03-26T21:49:41.183754+0000","last_peered":"2024-03-26T21:49:41.183754+0000","last_clean":"2024-03-26T21:49:41.183754+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:49:41.183754+0000","last_undegraded":"2024-03-26T21:49:41.183754+0000","last_fullsized":"2024-03-26T21:49:41.183754+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.024681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.024681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:00.912970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":902,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178858+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:49:41.178858+0000","last_peered":"2024-03-26T21:49:41.178858+0000","last_clean":"2024-03-26T21:49:41.178858+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:49:41.178858+0000","last_undegraded":"2024-03-26T21:49:41.178858+0000","last_fullsized":"2024-03-26T21:49:41.178858+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":579,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179688+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:49:41.179688+0000","last_peered":"2024-03-26T21:49:41.179688+0000","last_clean":"2024-03-26T21:49:41.179688+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:49:41.179688+0000","last_undegraded":"2024-03-26T21:49:41.179688+0000","last_fullsized":"2024-03-26T21:49:41.179688+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:43.666911+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":614,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178619+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:49:41.178619+0000","last_peered":"2024-03-26T21:49:41.178619+0000","last_clean":"2024-03-26T21:49:41.178619+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:49:41.178619+0000","last_undegraded":"2024-03-26T21:49:41.178619+0000","last_fullsized":"2024-03-26T21:49:41.178619+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":908,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174140+0000","last_change":"2024-03-26T21:48:29.849767+0000","last_active":"2024-03-26T21:49:41.174140+0000","last_peered":"2024-03-26T21:49:41.174140+0000","last_clean":"2024-03-26T21:49:41.174140+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:49:41.174140+0000","last_undegraded":"2024-03-26T21:49:41.174140+0000","last_fullsized":"2024-03-26T21:49:41.174140+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.849687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.849687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:58.749020+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":548,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.168717+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:49:41.168717+0000","last_peered":"2024-03-26T21:49:41.168717+0000","last_clean":"2024-03-26T21:49:41.168717+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:49:41.168717+0000","last_undegraded":"2024-03-26T21:49:41.168717+0000","last_fullsized":"2024-03-26T21:49:41.168717+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":895,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.172873+0000","last_change":"2024-03-26T21:49:33.717279+0000","last_active":"2024-03-26T21:49:41.172873+0000","last_peered":"2024-03-26T21:49:41.172873+0000","last_clean":"2024-03-26T21:49:41.172873+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:49:41.172873+0000","last_undegraded":"2024-03-26T21:49:41.172873+0000","last_fullsized":"2024-03-26T21:49:41.172873+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:33.716995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:49:33.716995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:48.354109+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":646,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178205+0000","last_change":"2024-03-26T21:48:27.673802+0000","last_active":"2024-03-26T21:49:41.178205+0000","last_peered":"2024-03-26T21:49:41.178205+0000","last_clean":"2024-03-26T21:49:41.178205+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:49:41.178205+0000","last_undegraded":"2024-03-26T21:49:41.178205+0000","last_fullsized":"2024-03-26T21:49:41.178205+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:27.673516+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.673516+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:55.239856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":908,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174563+0000","last_change":"2024-03-26T21:48:34.769285+0000","last_active":"2024-03-26T21:49:41.174563+0000","last_peered":"2024-03-26T21:49:41.174563+0000","last_clean":"2024-03-26T21:49:41.174563+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:49:41.174563+0000","last_undegraded":"2024-03-26T21:49:41.174563+0000","last_fullsized":"2024-03-26T21:49:41.174563+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:34.769168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.769168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":888,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.173602+0000","last_change":"2024-03-26T21:48:30.838758+0000","last_active":"2024-03-26T21:49:41.173602+0000","last_peered":"2024-03-26T21:49:41.173602+0000","last_clean":"2024-03-26T21:49:41.173602+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:49:41.173602+0000","last_undegraded":"2024-03-26T21:49:41.173602+0000","last_fullsized":"2024-03-26T21:49:41.173602+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.838637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:30.838637+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.838637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:57.825792+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":870,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.181745+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:49:41.181745+0000","last_peered":"2024-03-26T21:49:41.181745+0000","last_clean":"2024-03-26T21:49:41.181745+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:49:41.181745+0000","last_undegraded":"2024-03-26T21:49:41.181745+0000","last_fullsized":"2024-03-26T21:49:41.181745+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":289,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.177932+0000","last_change":"2024-03-26T21:48:59.006965+0000","last_active":"2024-03-26T21:49:41.177932+0000","last_peered":"2024-03-26T21:49:41.177932+0000","last_clean":"2024-03-26T21:49:41.177932+0000","last_became_active":"2024-03-26T21:48:58.977997+0000","last_became_peered":"2024-03-26T21:48:58.977997+0000","last_unstale":"2024-03-26T21:49:41.177932+0000","last_undegraded":"2024-03-26T21:49:41.177932+0000","last_fullsized":"2024-03-26T21:49:41.177932+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:34.910272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.910272+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:58.805574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":239,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.174455+0000","last_change":"2024-03-26T21:48:58.976105+0000","last_active":"2024-03-26T21:49:41.174455+0000","last_peered":"2024-03-26T21:49:41.174455+0000","last_clean":"2024-03-26T21:49:41.174455+0000","last_became_active":"2024-03-26T21:48:58.975714+0000","last_became_peered":"2024-03-26T21:48:58.975714+0000","last_unstale":"2024-03-26T21:49:41.174455+0000","last_undegraded":"2024-03-26T21:49:41.174455+0000","last_fullsized":"2024-03-26T21:49:41.174455+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":578,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179276+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:49:41.179276+0000","last_peered":"2024-03-26T21:49:41.179276+0000","last_clean":"2024-03-26T21:49:41.179276+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:49:41.179276+0000","last_undegraded":"2024-03-26T21:49:41.179276+0000","last_fullsized":"2024-03-26T21:49:41.179276+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":232,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176824+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:49:41.176824+0000","last_peered":"2024-03-26T21:49:41.176824+0000","last_clean":"2024-03-26T21:49:41.176824+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:49:41.176824+0000","last_undegraded":"2024-03-26T21:49:41.176824+0000","last_fullsized":"2024-03-26T21:49:41.176824+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":622,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179402+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:49:41.179402+0000","last_peered":"2024-03-26T21:49:41.179402+0000","last_clean":"2024-03-26T21:49:41.179402+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:49:41.179402+0000","last_undegraded":"2024-03-26T21:49:41.179402+0000","last_fullsized":"2024-03-26T21:49:41.179402+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":889,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.179471+0000","last_change":"2024-03-26T21:48:58.978607+0000","last_active":"2024-03-26T21:49:41.179471+0000","last_peered":"2024-03-26T21:49:41.179471+0000","last_clean":"2024-03-26T21:49:41.179471+0000","last_became_active":"2024-03-26T21:48:58.978253+0000","last_became_peered":"2024-03-26T21:48:58.978253+0000","last_unstale":"2024-03-26T21:49:41.179471+0000","last_undegraded":"2024-03-26T21:49:41.179471+0000","last_fullsized":"2024-03-26T21:49:41.179471+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:33.984501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.984501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":890,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.176667+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:49:41.176667+0000","last_peered":"2024-03-26T21:49:41.176667+0000","last_clean":"2024-03-26T21:49:41.176667+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:49:41.176667+0000","last_undegraded":"2024-03-26T21:49:41.176667+0000","last_fullsized":"2024-03-26T21:49:41.176667+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":586,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-03-26T21:49:41.178109+0000","last_change":"2024-03-26T21:48:44.829403+0000","last_active":"2024-03-26T21:49:41.178109+0000","last_peered":"2024-03-26T21:49:41.178109+0000","last_clean":"2024-03-26T21:49:41.178109+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:49:41.178109+0000","last_undegraded":"2024-03-26T21:49:41.178109+0000","last_fullsized":"2024-03-26T21:49:41.178109+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.829181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.829181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:49:56.692365+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6596395032,"num_objects":100654,"num_object_clones":0,"num_object_copies":301962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":17329,"num_objects_unfound":0,"num_objects_dirty":100654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136586,"num_write_kb":7591617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21600,"num_bytes_recovered":1415577600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21163356160,"data_stored":21163343944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74514,"ondisk_log_size":74514,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1147,"num_read_kb":1142,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":7130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1072,"ondisk_log_size":1072,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973797,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3079816,"kb_used_data":2557056,"kb_used_omap":6,"kb_used_meta":522745,"kb_avail":90665336,"statfs":{"total":95995035648,"available":92841304064,"internally_reserved":0,"allocated":2618425344,"data_stored":2618003369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":535291176},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365064,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3931524,"kb_used_data":3264504,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":89813628,"statfs":{"total":95995035648,"available":91969155072,"internally_reserved":0,"allocated":3342852096,"data_stored":3342436393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":683014598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575138,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4494872,"kb_used_data":3583504,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":89250280,"statfs":{"total":95995035648,"available":91392286720,"internally_reserved":0,"allocated":3669508096,"data_stored":3669068558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738663,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3282836,"kb_used_data":2399244,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90462316,"statfs":{"total":95995035648,"available":92633411584,"internally_reserved":0,"allocated":2456825856,"data_stored":2456391229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149974,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3768272,"kb_used_data":2834696,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":89976880,"statfs":{"total":95995035648,"available":92136325120,"internally_reserved":0,"allocated":2902728704,"data_stored":2902296855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":955972039},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650570,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2797504,"kb_used_data":2030968,"kb_used_omap":4,"kb_used_meta":766523,"kb_avail":90947648,"statfs":{"total":95995035648,"available":93130391552,"internally_reserved":0,"allocated":2079711232,"data_stored":2079289373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":784920227},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659403,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2353472,"kb_used_data":1636468,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":91391680,"statfs":{"total":95995035648,"available":93585080320,"internally_reserved":0,"allocated":1675743232,"data_stored":1675331383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2446589952,"data_stored":2446589952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3661434880,"data_stored":3661430808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2487943168,"data_stored":2487943168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2889154560,"data_stored":2889154560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2072383488,"data_stored":2072379416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1667563520,"data_stored":1667563520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3329953792,"data_stored":3329949720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2608332800,"data_stored":2608332800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:50:02.624 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-26T21:50:02.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:02 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:50:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:02 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:50:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1337/266196 objects degraded (0.502%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:50:02.750 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:50:02.750 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:50:02.751 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1337/266196 objects degraded (0.502%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:50:02.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:50:02.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:50:02.751 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1337/266196 objects degraded (0.502%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T21:50:02.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:02.816 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:02.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:02.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:03.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:03.199 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:03.236 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:03.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.374 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:03.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:03.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:03.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:03 smithi055 ceph-mon[29406]: pgmap v1022: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 740 op/s; 1426/264726 objects degraded (0.539%); 11068/264726 objects misplaced (4.181%); 16 MiB/s, 258 objects/s recovering 2024-03-26T21:50:03.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:03 smithi055 ceph-mon[40587]: pgmap v1022: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 740 op/s; 1426/264726 objects degraded (0.539%); 11068/264726 objects misplaced (4.181%); 16 MiB/s, 258 objects/s recovering 2024-03-26T21:50:03.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:03.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:03.653 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:03 smithi067 ceph-mon[33346]: pgmap v1022: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 740 op/s; 1426/264726 objects degraded (0.539%); 11068/264726 objects misplaced (4.181%); 16 MiB/s, 258 objects/s recovering 2024-03-26T21:50:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:03.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:03.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:03.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:03.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:04.030 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.071 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:04.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:04.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:04.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:04.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:04.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:04.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:04.577 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:04.611 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:04.713 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.751 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:04.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:04.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:04 smithi055 ceph-mon[29406]: pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 1629/252747 objects degraded (0.645%); 9998/252747 objects misplaced (3.956%); 19 MiB/s, 297 objects/s recovering 2024-03-26T21:50:04.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:04 smithi055 ceph-mon[40587]: pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 1629/252747 objects degraded (0.645%); 9998/252747 objects misplaced (3.956%); 19 MiB/s, 297 objects/s recovering 2024-03-26T21:50:04.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:04.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:04.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:04 smithi067 ceph-mon[33346]: pgmap v1023: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 1629/252747 objects degraded (0.645%); 9998/252747 objects misplaced (3.956%); 19 MiB/s, 297 objects/s recovering 2024-03-26T21:50:04.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:05.137 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:05.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:05.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:05.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.458 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:05.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:05.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.601 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:05.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:05.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:05.736 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:166860 9.8 e278/276 9:1833cc5c:::benchmark_data_smithi055_135863_object36209:head v 278'10504, mlcod=277'10501)", 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:26.813260+0000", 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 519.01584351300005, 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.24892710800000001, 2024-03-26T21:50:05.825 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813260+0000", 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813260+0000", 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.826 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813261+0000", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1519999999999999e-06 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813268+0000", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2199999999999997e-06 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813280+0000", 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2187999999999999e-05 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.827 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813287+0000", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3719999999999999e-06 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813389+0000", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010167599999999999 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.813411+0000", 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1789000000000001e-05 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.828 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.062127+0000", 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.24871687200000001 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.062187+0000", 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9839000000000001e-05 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:166872 9.1 e278/276 9:8abb16dd:::benchmark_data_smithi055_135863_object36221:head v 278'10488, mlcod=278'10486)", 2024-03-26T21:50:05.829 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:26.820990+0000", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 519.00811382999996, 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.24907453600000001, 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.820990+0000", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.820990+0000", 2024-03-26T21:50:05.830 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.820991+0000", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.55e-06 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.821000+0000", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3229999999999998e-06 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.821014+0000", 2024-03-26T21:50:05.831 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3939e-05 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.821023+0000", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3859999999999996e-06 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.821098+0000", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4540000000000001e-05 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.821120+0000", 2024-03-26T21:50:05.832 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.2388999999999999e-05 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.070025+0000", 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.248904551 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.070064+0000", 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9857999999999997e-05 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.833 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:166874 9.1 e278/276 9:857252d8:::benchmark_data_smithi055_135863_object36223:head v 278'10489, mlcod=278'10486)", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:26.824377+0000", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 519.004726925, 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.24571688899999999, 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824377+0000", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.834 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824377+0000", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824377+0000", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.7500000000000001e-07 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824381+0000", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.177e-06 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.835 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824387+0000", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7440000000000004e-06 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.824391+0000", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3089999999999997e-06 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.877279+0000", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.052887888000000001 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.836 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.877306+0000", 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.6509999999999999e-05 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.070069+0000", 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.19276312700000001 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.070094+0000", 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4759000000000001e-05 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.837 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.838 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.838 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_repop(client.25220.0:166876 9.8 e278/276 9:12d38f9d:::benchmark_data_smithi055_135863_object36225:head v 278'10506, mlcod=278'10502)", 2024-03-26T21:50:05.838 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:41:26.826341+0000", 2024-03-26T21:50:05.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:05.839 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 519.00276313400002, 2024-03-26T21:50:05.839 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.244753049, 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826341+0000", 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826341+0000", 2024-03-26T21:50:05.840 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826342+0000", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.161e-06 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826349+0000", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.4499999999999998e-06 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826364+0000", 2024-03-26T21:50:05.841 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4654999999999999e-05 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.826372+0000", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2029999999999992e-06 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.932882+0000", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.106510069 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:26.932910+0000", 2024-03-26T21:50:05.842 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7594e-05 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.071011+0000", 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.138101433 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:41:27.071094+0000", 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2484000000000005e-05 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:112166 10.15 10:afa930a3:::benchmark_data_smithi055_163313_object8117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:21.544039+0000", 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 284.28506430300001, 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.34634719000000003, 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 112166 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.844 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544039+0000", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544039+0000", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544041+0000", 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.463e-06 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.845 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544047+0000", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9569999999999999e-06 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544060+0000", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2814e-05 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544068+0000", 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2900000000000002e-06 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.846 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544244+0000", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00017579600000000001 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544271+0000", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.7679e-05 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.862451+0000", 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.31817991600000001 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.847 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.862564+0000", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112561 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.862879+0000", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000315474 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870074+0000", 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0071949350000000004 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.848 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.874957+0000", 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0048824740000000004 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890255+0000", 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.015298714 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.849 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890333+0000", 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7708999999999997e-05 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890387+0000", 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3408000000000001e-05 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:112169 10.15 10:ac2f0544:::benchmark_data_smithi055_163313_object8120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:21.544088+0000", 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 284.28501550300001, 2024-03-26T21:50:05.850 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.346501956, 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 112169 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544088+0000", 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.851 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544088+0000", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544089+0000", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6000000000000004e-07 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544092+0000", 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9730000000000002e-06 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.852 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544098+0000", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9920000000000004e-06 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544104+0000", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9889999999999997e-06 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544304+0000", 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000199774 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.853 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.544313+0000", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7599999999999997e-06 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.865181+0000", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.32086763800000001 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.865314+0000", 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013352500000000001 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.865610+0000", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029577500000000002 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870183+0000", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0045730379999999998 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.875034+0000", 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0048507050000000003 2024-03-26T21:50:05.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890438+0000", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.015404007000000001 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890554+0000", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011560799999999999 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890590+0000", 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6612000000000001e-05 2024-03-26T21:50:05.856 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:112182 10.15 10:addbbbf4:::benchmark_data_smithi055_163313_object8133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:21.554010+0000", 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 284.275093755, 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.336725414, 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:05.857 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 112182 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554010+0000", 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554010+0000", 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.858 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554012+0000", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.826e-06 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554020+0000", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4160000000000004e-06 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554047+0000", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.6991999999999998e-05 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.859 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554057+0000", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7699999999999996e-06 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554137+0000", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0178000000000004e-05 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.554155+0000", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7604999999999998e-05 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.860 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.867090+0000", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.31293515599999999 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.867216+0000", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012561000000000001 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.867555+0000", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00033964200000000003 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.861 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870229+0000", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002673581 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.875039+0000", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0048100840000000001 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890626+0000", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.015587316 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.862 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890694+0000", 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7459999999999994e-05 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890735+0000", 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1777999999999999e-05 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:112195 10.15 10:aff34cee:::benchmark_data_smithi055_163313_object8146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:21.566586+0000", 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 284.262518, 2024-03-26T21:50:05.863 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.32428188299999999, 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 112195 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566586+0000", 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.864 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566586+0000", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566588+0000", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0310000000000001e-06 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566595+0000", 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.0389999999999999e-06 2024-03-26T21:50:05.865 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566612+0000", 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7226000000000001e-05 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.867 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.868 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566622+0000", 2024-03-26T21:50:05.868 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0035e-05 2024-03-26T21:50:05.868 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.868 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.868 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566931+0000", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00030917100000000001 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.566944+0000", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2683999999999999e-05 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870262+0000", 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.30331785300000003 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.869 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870389+0000", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012765699999999999 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.870689+0000", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00029940499999999998 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.874936+0000", 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0042473809999999997 2024-03-26T21:50:05.870 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.875044+0000", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00010778299999999999 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890774+0000", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.015729918999999998 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890822+0000", 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7712e-05 2024-03-26T21:50:05.871 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890868+0000", 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.5986999999999999e-05 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:112228 10.15 10:aa321a7d:::benchmark_data_smithi055_163313_object8179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:45:21.600108+0000", 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 284.228996177, 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.29090554299999999, 2024-03-26T21:50:05.872 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 112228 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600108+0000", 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.873 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600108+0000", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600109+0000", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.446e-06 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600115+0000", 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2110000000000003e-06 2024-03-26T21:50:05.874 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600131+0000", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5967000000000001e-05 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600140+0000", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6740000000000003e-06 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600212+0000", 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2111000000000001e-05 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.600223+0000", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.147e-05 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.873081+0000", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.27285804299999999 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.873207+0000", 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012566500000000001 2024-03-26T21:50:05.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,5", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.873482+0000", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00027482800000000002 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.879759+0000", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0062772269999999998 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.880441+0000", 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00068203899999999995 2024-03-26T21:50:05.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890917+0000", 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.010476022999999999 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.890970+0000", 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.3139000000000001e-05 2024-03-26T21:50:05.878 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:45:21.891013+0000", 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.2700000000000001e-05 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125011 11.1 11:88f3bee3:::benchmark_data_smithi055_186849_object6389:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:28.976603+0000", 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.852500370000001, 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.017477728, 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.879 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125011 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976603+0000", 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.880 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976603+0000", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976605+0000", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6330000000000001e-06 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976609+0000", 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9199999999999997e-06 2024-03-26T21:50:05.881 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976619+0000", 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5999999999999996e-06 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976628+0000", 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5980000000000003e-06 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.882 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976878+0000", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00024965499999999999 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976888+0000", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0467e-05 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.989590+0000", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.01270174 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.883 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.994081+0000", 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0044911150000000004 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125023 11.1 11:83983cb9:::benchmark_data_smithi055_186849_object6401:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.852461812000001, 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0571867800000001, 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.884 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125023 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.885 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976642+0000", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1399999999999997e-07 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976645+0000", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9809999999999999e-06 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976651+0000", 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0689999999999998e-06 2024-03-26T21:50:05.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976657+0000", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8810000000000001e-06 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976915+0000", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025755500000000002 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:28.976923+0000", 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9950000000000005e-06 2024-03-26T21:50:05.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.888 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.994612+0000", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0176892049999999 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.995379+0000", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00076701899999999997 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.995859+0000", 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00047994200000000002 2024-03-26T21:50:05.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.012425+0000", 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.016566185000000001 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.012465+0000", 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.9740000000000002e-05 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.890 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.032767+0000", 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.020301514999999999 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.033771+0000", 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0010042689999999999 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.033829+0000", 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.791e-05 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125044 11.1 11:89c47966:::benchmark_data_smithi055_186849_object6422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.821596061999998, 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0269018089999999, 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125044 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.892 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007508+0000", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007509+0000", 2024-03-26T21:50:05.893 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.35e-06 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007513+0000", 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3080000000000001e-06 2024-03-26T21:50:05.894 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007521+0000", 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6450000000000005e-06 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.007527+0000", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.947e-06 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.008029+0000", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00050181200000000003 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.008040+0000", 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.092e-05 2024-03-26T21:50:05.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.004736+0000", 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99669598100000001 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.005174+0000", 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000438732 2024-03-26T21:50:05.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.005313+0000", 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000138775 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.012501+0000", 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0071879759999999996 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025019+0000", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012518008000000001 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.033885+0000", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0088658179999999993 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034361+0000", 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00047576200000000002 2024-03-26T21:50:05.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034409+0000", 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8775e-05 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125046 11.1 11:8032de69:::benchmark_data_smithi055_186849_object6424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.817424265, 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0229080290000001, 2024-03-26T21:50:05.900 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125046 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.901 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011679+0000", 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011681+0000", 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.477e-06 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.902 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011686+0000", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2290000000000002e-06 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011696+0000", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0325e-05 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.011705+0000", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0600000000000008e-06 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.903 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.012054+0000", 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00034953000000000002 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.012064+0000", 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.027e-05 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.008723+0000", 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99665908000000003 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.904 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.008844+0000", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012101599999999999 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.008967+0000", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012228799999999999 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.012538+0000", 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003570976 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.905 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025059+0000", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012521302999999999 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034419+0000", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0093598989999999996 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034540+0000", 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012112600000000001 2024-03-26T21:50:05.906 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034587+0000", 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7450000000000001e-05 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125055 11.1 11:886d7e96:::benchmark_data_smithi055_186849_object6433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.808565563999998, 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.017184372, 2024-03-26T21:50:05.907 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125055 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.908 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020538+0000", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020540+0000", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.508e-06 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020545+0000", 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1680000000000003e-06 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.909 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020556+0000", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.155e-05 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020565+0000", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4090000000000003e-06 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020639+0000", 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3950999999999997e-05 2024-03-26T21:50:05.910 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.911 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.911 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.911 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.020649+0000", 2024-03-26T21:50:05.912 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9420000000000007e-06 2024-03-26T21:50:05.912 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.912 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.012566+0000", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99191706899999998 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.013240+0000", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00067413599999999998 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.013381+0000", 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014065900000000001 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.913 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025093+0000", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011712353 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025169+0000", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6588000000000006e-05 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.034597+0000", 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00942723 2024-03-26T21:50:05.914 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037678+0000", 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00308144 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037723+0000", 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4369000000000002e-05 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.915 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125061 11.1 11:87008943:::benchmark_data_smithi055_186849_object6439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.803806913000003, 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.012501627, 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125061 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.916 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025297+0000", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025298+0000", 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.299e-06 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.917 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025304+0000", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3339999999999998e-06 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025314+0000", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3579999999999992e-06 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025322+0000", 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2609999999999993e-06 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.918 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025407+0000", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.5407000000000004e-05 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.025418+0000", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0338e-05 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.017574+0000", 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99215618299999997 2024-03-26T21:50:05.919 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.017695+0000", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000121131 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.017819+0000", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00012357199999999999 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025128+0000", 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0073095240000000004 2024-03-26T21:50:05.920 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.033874+0000", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087453949999999996 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037735+0000", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038610979999999999 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037769+0000", 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4039999999999999e-05 2024-03-26T21:50:05.921 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037798+0000", 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9686999999999999e-05 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125063 11.1 11:899c3b6b:::benchmark_data_smithi055_186849_object6441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.8023126, 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.011160453, 2024-03-26T21:50:05.922 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125063 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.923 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026791+0000", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026792+0000", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.249e-06 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026799+0000", 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8539999999999999e-06 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.924 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026815+0000", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5523e-05 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026823+0000", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3010000000000007e-06 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026907+0000", 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4260999999999998e-05 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.925 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.026917+0000", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9299999999999998e-06 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.021063+0000", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.99414525899999995 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.021196+0000", 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013394000000000001 2024-03-26T21:50:05.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.021332+0000", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013600099999999999 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025204+0000", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0038712379999999999 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037804+0000", 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012600264 2024-03-26T21:50:05.927 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037885+0000", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1394e-05 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037922+0000", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6307999999999999e-05 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037952+0000", 2024-03-26T21:50:05.928 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9930999999999999e-05 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125093 11.1 11:8647a194:::benchmark_data_smithi055_186849_object6471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.762679857000002, 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.97162580700000001, 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.929 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125093 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066424+0000", 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.930 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066425+0000", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.127e-06 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066431+0000", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.348e-06 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066446+0000", 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.467e-05 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.931 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066454+0000", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8830000000000006e-06 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066540+0000", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6306999999999999e-05 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.066550+0000", 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7520000000000008e-06 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.932 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025229+0000", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.958678892 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025342+0000", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113454 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.025456+0000", 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113576 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.933 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037809+0000", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012352762 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037852+0000", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.3684999999999999e-05 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037983+0000", 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00013054399999999999 2024-03-26T21:50:05.934 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038020+0000", 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6677000000000001e-05 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038050+0000", 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0130000000000001e-05 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.935 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125100 11.1 11:85fc4cb9:::benchmark_data_smithi055_186849_object6478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.747523665000003, 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.956623377, 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125100 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.936 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081580+0000", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081582+0000", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8530000000000001e-06 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081588+0000", 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5139999999999998e-06 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081602+0000", 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3356000000000001e-05 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.081610+0000", 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.6959999999999994e-06 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.938 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.082340+0000", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00072962600000000004 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.082353+0000", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2415000000000001e-05 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.029285+0000", 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94693205700000005 2024-03-26T21:50:05.939 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.029621+0000", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00033691199999999998 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.029752+0000", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000130216 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.037813+0000", 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0080613769999999998 2024-03-26T21:50:05.940 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038084+0000", 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00027087699999999999 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038117+0000", 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3374000000000003e-05 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.941 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038173+0000", 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.5544999999999998e-05 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038203+0000", 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0559000000000003e-05 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.942 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125111 11.1 11:8caa63c6:::benchmark_data_smithi055_186849_object6489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.734176734000002, 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.95817205699999997, 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125111 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.943 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094927+0000", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094928+0000", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8400000000000004e-07 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.944 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094932+0000", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0640000000000004e-06 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094940+0000", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4470000000000006e-06 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.094947+0000", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.443e-06 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.945 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.097012+0000", 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020651699999999999 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.097022+0000", 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0214000000000001e-05 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038237+0000", 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.941214573 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.946 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038350+0000", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011338899999999999 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.038464+0000", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011374699999999999 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.049842+0000", 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.011378249 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.947 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.052878+0000", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030355680000000002 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.052924+0000", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6097999999999998e-05 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.052970+0000", 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6300000000000001e-05 2024-03-26T21:50:05.948 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.053099+0000", 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000129111 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15804.0:125122 11.1 11:807e46d6:::benchmark_data_smithi055_186849_object6500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 36.695326526000002, 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92253444399999995, 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:50:05.949 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15804", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 125122 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:50:05.950 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133777+0000", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133778+0000", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1009999999999999e-06 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133783+0000", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0030000000000002e-06 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:50:05.951 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133793+0000", 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9289999999999993e-06 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133799+0000", 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0639999999999999e-06 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133870+0000", 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1194999999999999e-05 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.952 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:29.133883+0000", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2337999999999999e-05 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.041539+0000", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90765587199999997 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.041653+0000", 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000113887 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.953 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 0,6", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.041767+0000", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011404999999999999 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.049918+0000", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081511220000000002 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.052884+0000", 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.002966677 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.954 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.056231+0000", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0033464689999999999 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.056276+0000", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4928000000000001e-05 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:49:30.056312+0000", 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5809000000000002e-05 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.955 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.956 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.956 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:05.956 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:05.956 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:50:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:05.957 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:05.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:06.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:06.098 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:06.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:06.238 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:06.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:06.375 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:06.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:06.615 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:06.654 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:06.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:06.792 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:06.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:06.929 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:06.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:07.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:07.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:06 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:50:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:06 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:50:07.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:06 smithi067 ceph-mon[33346]: pgmap v1024: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 1629/251892 objects degraded (0.647%); 9998/251892 objects misplaced (3.969%); 16 MiB/s, 262 objects/s recovering 2024-03-26T21:50:07.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[29406]: pgmap v1024: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 1629/251892 objects degraded (0.647%); 9998/251892 objects misplaced (3.969%); 16 MiB/s, 262 objects/s recovering 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:50:07.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:06 smithi055 ceph-mon[40587]: pgmap v1024: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 946 op/s; 1629/251892 objects degraded (0.647%); 9998/251892 objects misplaced (3.969%); 16 MiB/s, 262 objects/s recovering 2024-03-26T21:50:07.341 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:07.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:50:07.443 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:07.480 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:07.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.618 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:07.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:07.625 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:50:07.625 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:50:07.625 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T21:50:07.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:50:07.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:07.770 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:07.872 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:07.926 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:07.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:07.978 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:50:07.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:07.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:50:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:50:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: 5.1b deep-scrub starts 2024-03-26T21:50:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: 5.1b deep-scrub ok 2024-03-26T21:50:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: osdmap e479: 8 total, 8 up, 7 in 2024-03-26T21:50:07.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:07 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1629/251892 objects degraded (0.647%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:08.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: 5.1b deep-scrub starts 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: 5.1b deep-scrub ok 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: osdmap e479: 8 total, 8 up, 7 in 2024-03-26T21:50:08.030 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1629/251892 objects degraded (0.647%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:08.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:50:08.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:50:08.031 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: 5.1b deep-scrub starts 2024-03-26T21:50:08.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: 5.1b deep-scrub ok 2024-03-26T21:50:08.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: osdmap e479: 8 total, 8 up, 7 in 2024-03-26T21:50:08.032 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:07 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1629/251892 objects degraded (0.647%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:08.058 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:08.112 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:08.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:08.193 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:08.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:08.321 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:08.423 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:08.462 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:08.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.608 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:08.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:08.708 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:08.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:08.785 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:08.888 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:08.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:08.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:08 smithi067 ceph-mon[33346]: pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 616 B/s rd, 0 B/s wr, 683 op/s; 1775/249165 objects degraded (0.712%); 9712/249165 objects misplaced (3.898%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T21:50:09.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.095 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:09.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:09.165 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:09.165 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:08 smithi055 ceph-mon[40587]: pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 616 B/s rd, 0 B/s wr, 683 op/s; 1775/249165 objects degraded (0.712%); 9712/249165 objects misplaced (3.898%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T21:50:09.166 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:09.166 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:08 smithi055 ceph-mon[29406]: pgmap v1026: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 616 B/s rd, 0 B/s wr, 683 op/s; 1775/249165 objects degraded (0.712%); 9712/249165 objects misplaced (3.898%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T21:50:09.166 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:09.197 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:09.245 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:09.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:09.384 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:09.486 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:09.529 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:09.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:09.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:09.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:09.805 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:09.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:09.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:09.960 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:50:10.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:10.077 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:10.180 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:10.230 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:09 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:10.232 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:10.309 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:09 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:10.310 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:09 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:10.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.379 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:10.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:10.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:10.489 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=cleanup t=2024-03-26T21:50:10.319916344Z level=info msg="Completed cleanup jobs" duration=6.93974ms 2024-03-26T21:50:10.518 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:10.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:10.740 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:50:10.740 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:50:10.740 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-26T21:50:10.740 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:50:10.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:10.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:10.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:10.942 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:10.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:11.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:11.060 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:50:11.060 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:50:11.061 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: pgmap v1027: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 983 op/s; 2003/236706 objects degraded (0.846%); 8656/236706 objects misplaced (3.657%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:50:11.061 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:11.061 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:11.061 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:10 smithi067 ceph-mon[33346]: osdmap e480: 8 total, 8 up, 7 in 2024-03-26T21:50:11.081 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:11.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.214 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:11.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:50:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:50:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: pgmap v1027: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 983 op/s; 2003/236706 objects degraded (0.846%); 8656/236706 objects misplaced (3.657%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:50:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:11.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[40587]: osdmap e480: 8 total, 8 up, 7 in 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: pgmap v1027: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 983 op/s; 2003/236706 objects degraded (0.846%); 8656/236706 objects misplaced (3.657%); 11 MiB/s, 178 objects/s recovering 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4142427486' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:10 smithi055 ceph-mon[29406]: osdmap e480: 8 total, 8 up, 7 in 2024-03-26T21:50:11.354 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:11.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:11.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:11.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:11.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:50:11.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.757 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:11.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:11.898 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:11.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:11.966 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:11 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2003/236706 objects degraded (0.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:12.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.055 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:12.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:12.156 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:12.199 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:12.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:12.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:11 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2003/236706 objects degraded (0.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:12.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:11 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2003/236706 objects degraded (0.846%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:12.336 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:12.438 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.476 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:12.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:12.578 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:12.606 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:12.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:12.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:12.878 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:12.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:12.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:13.025 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:13.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:12 smithi067 ceph-mon[33346]: pgmap v1029: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 2166/234861 objects degraded (0.922%); 8362/234861 objects misplaced (3.560%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:50:13.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:13.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:12 smithi055 ceph-mon[40587]: pgmap v1029: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 2166/234861 objects degraded (0.922%); 8362/234861 objects misplaced (3.560%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:50:13.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:12 smithi055 ceph-mon[29406]: pgmap v1029: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 2166/234861 objects degraded (0.922%); 8362/234861 objects misplaced (3.560%); 8.0 MiB/s, 128 objects/s recovering 2024-03-26T21:50:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:13.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:13.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:13.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:13.816 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:50:13.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:13.824 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:50:13.825 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:13.871 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:13.911 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:13.951 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:13.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:13.993 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:13.999 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:13.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:14.036 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:14.069 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1239574599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:14.074 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:14.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:14.118 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:14.118 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1239574599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:14.118 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:13 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1239574599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:14.125 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:14.136 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:14.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:14.382 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:14.420 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:14.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:14.587 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:14.587 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:14.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:14.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:50:14.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:14.830 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:14.886 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2024-03-26T21:50:13.766721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4853995968,"num_objects":74256,"num_object_clones":0,"num_object_copies":222768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2415,"num_objects_misplaced":7334,"num_objects_unfound":0,"num_objects_dirty":74256,"num_whiteouts":0,"num_read":2920,"num_read_kb":2768,"num_write":164369,"num_write_kb":7594078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29253,"num_bytes_recovered":1894524122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76027,"ondisk_log_size":76027,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19535752,"kb_used_data":13962888,"kb_used_omap":18,"kb_used_meta":5572525,"kb_avail":636680312,"statfs":{"total":671965249536,"available":651960639488,"internally_reserved":0,"allocated":14297997312,"data_stored":14295018248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19289,"internal_metadata":5706265767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,8,4,7,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-636223488,"num_objects":-9708,"num_object_clones":0,"num_object_copies":-29124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":-2664,"num_objects_unfound":0,"num_objects_dirty":-9708,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9708,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003344"},"pg_stats":[{"pgid":"11.3","version":"480'10186","reported_seq":10300,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.461177+0000","last_change":"2024-03-26T21:49:29.997467+0000","last_active":"2024-03-26T21:50:13.461177+0000","last_peered":"2024-03-26T21:50:13.461177+0000","last_clean":"2024-03-26T21:50:13.461177+0000","last_became_active":"2024-03-26T21:49:29.980238+0000","last_became_peered":"2024-03-26T21:49:29.980238+0000","last_unstale":"2024-03-26T21:50:13.461177+0000","last_undegraded":"2024-03-26T21:50:13.461177+0000","last_fullsized":"2024-03-26T21:50:13.461177+0000","mapping_epoch":467,"log_start":"456'5477","ondisk_log_start":"456'5477","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":99,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858816,"num_objects":4606,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10186,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'35","reported_seq":694,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965735+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:50:09.965735+0000","last_peered":"2024-03-26T21:50:09.965735+0000","last_clean":"2024-03-26T21:50:09.965735+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:50:09.965735+0000","last_undegraded":"2024-03-26T21:50:09.965735+0000","last_fullsized":"2024-03-26T21:50:09.965735+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975435+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:50:09.975435+0000","last_peered":"2024-03-26T21:50:09.975435+0000","last_clean":"2024-03-26T21:50:09.975435+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:50:09.975435+0000","last_undegraded":"2024-03-26T21:50:09.975435+0000","last_fullsized":"2024-03-26T21:50:09.975435+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"462'12","reported_seq":709,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.973343+0000","last_change":"2024-03-26T21:48:59.006658+0000","last_active":"2024-03-26T21:50:09.973343+0000","last_peered":"2024-03-26T21:50:09.973343+0000","last_clean":"2024-03-26T21:50:09.973343+0000","last_became_active":"2024-03-26T21:48:58.995625+0000","last_became_peered":"2024-03-26T21:48:58.995625+0000","last_unstale":"2024-03-26T21:50:09.973343+0000","last_undegraded":"2024-03-26T21:50:09.973343+0000","last_fullsized":"2024-03-26T21:50:09.973343+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":604,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966513+0000","last_change":"2024-03-26T21:48:58.976822+0000","last_active":"2024-03-26T21:50:09.966513+0000","last_peered":"2024-03-26T21:50:09.966513+0000","last_clean":"2024-03-26T21:50:09.966513+0000","last_became_active":"2024-03-26T21:48:58.976172+0000","last_became_peered":"2024-03-26T21:48:58.976172+0000","last_unstale":"2024-03-26T21:50:09.966513+0000","last_undegraded":"2024-03-26T21:50:09.966513+0000","last_fullsized":"2024-03-26T21:50:09.966513+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"480'10306","reported_seq":13308,"reported_epoch":480,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:50:13.507540+0000","last_change":"2024-03-26T21:49:33.951482+0000","last_active":"2024-03-26T21:50:13.507540+0000","last_peered":"2024-03-26T21:50:13.507540+0000","last_clean":"2024-03-26T21:48:57.900582+0000","last_became_active":"2024-03-26T21:49:00.154091+0000","last_became_peered":"2024-03-26T21:49:00.154091+0000","last_unstale":"2024-03-26T21:50:13.507540+0000","last_undegraded":"2024-03-26T21:49:33.947342+0000","last_fullsized":"2024-03-26T21:48:59.963248+0000","mapping_epoch":462,"log_start":"458'5699","ondisk_log_start":"458'5699","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":849,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1395,"num_objects_misplaced":3507,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1442,"num_bytes_recovered":94502912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":3507},{"shards":"0,3","objects":-270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":690,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.973051+0000","last_change":"2024-03-26T21:48:59.000246+0000","last_active":"2024-03-26T21:50:09.973051+0000","last_peered":"2024-03-26T21:50:09.973051+0000","last_clean":"2024-03-26T21:50:09.973051+0000","last_became_active":"2024-03-26T21:48:58.980284+0000","last_became_peered":"2024-03-26T21:48:58.980284+0000","last_unstale":"2024-03-26T21:50:09.973051+0000","last_undegraded":"2024-03-26T21:50:09.973051+0000","last_fullsized":"2024-03-26T21:50:09.973051+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":362,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964798+0000","last_change":"2024-03-26T21:49:29.974892+0000","last_active":"2024-03-26T21:50:09.964798+0000","last_peered":"2024-03-26T21:50:09.964798+0000","last_clean":"2024-03-26T21:50:09.964798+0000","last_became_active":"2024-03-26T21:49:29.974562+0000","last_became_peered":"2024-03-26T21:49:29.974562+0000","last_unstale":"2024-03-26T21:50:09.964798+0000","last_undegraded":"2024-03-26T21:50:09.964798+0000","last_fullsized":"2024-03-26T21:50:09.964798+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":932,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974840+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:50:09.974840+0000","last_peered":"2024-03-26T21:50:09.974840+0000","last_clean":"2024-03-26T21:50:09.974840+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:50:09.974840+0000","last_undegraded":"2024-03-26T21:50:09.974840+0000","last_fullsized":"2024-03-26T21:50:09.974840+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":728,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975911+0000","last_change":"2024-03-26T21:48:58.974425+0000","last_active":"2024-03-26T21:50:09.975911+0000","last_peered":"2024-03-26T21:50:09.975911+0000","last_clean":"2024-03-26T21:50:09.975911+0000","last_became_active":"2024-03-26T21:48:58.974080+0000","last_became_peered":"2024-03-26T21:48:58.974080+0000","last_unstale":"2024-03-26T21:50:09.975911+0000","last_undegraded":"2024-03-26T21:50:09.975911+0000","last_fullsized":"2024-03-26T21:50:09.975911+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"480'10131","reported_seq":10241,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.473622+0000","last_change":"2024-03-26T21:49:29.994417+0000","last_active":"2024-03-26T21:50:13.473622+0000","last_peered":"2024-03-26T21:50:13.473622+0000","last_clean":"2024-03-26T21:50:13.473622+0000","last_became_active":"2024-03-26T21:49:29.979423+0000","last_became_peered":"2024-03-26T21:49:29.979423+0000","last_unstale":"2024-03-26T21:50:13.473622+0000","last_undegraded":"2024-03-26T21:50:13.473622+0000","last_fullsized":"2024-03-26T21:50:13.473622+0000","mapping_epoch":467,"log_start":"458'5566","ondisk_log_start":"458'5566","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297598976,"num_objects":4541,"num_object_clones":0,"num_object_copies":13623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":469504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":639,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966622+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:50:09.966622+0000","last_peered":"2024-03-26T21:50:09.966622+0000","last_clean":"2024-03-26T21:50:09.966622+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:50:09.966622+0000","last_undegraded":"2024-03-26T21:50:09.966622+0000","last_fullsized":"2024-03-26T21:50:09.966622+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":614,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966060+0000","last_change":"2024-03-26T21:48:58.979871+0000","last_active":"2024-03-26T21:50:09.966060+0000","last_peered":"2024-03-26T21:50:09.966060+0000","last_clean":"2024-03-26T21:50:09.966060+0000","last_became_active":"2024-03-26T21:48:58.979258+0000","last_became_peered":"2024-03-26T21:48:58.979258+0000","last_unstale":"2024-03-26T21:50:09.966060+0000","last_undegraded":"2024-03-26T21:50:09.966060+0000","last_fullsized":"2024-03-26T21:50:09.966060+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.750677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968329+0000","last_change":"2024-03-26T21:49:43.657948+0000","last_active":"2024-03-26T21:50:09.968329+0000","last_peered":"2024-03-26T21:50:09.968329+0000","last_clean":"2024-03-26T21:50:09.968329+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:50:09.968329+0000","last_undegraded":"2024-03-26T21:50:09.968329+0000","last_fullsized":"2024-03-26T21:50:09.968329+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:43.657872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:43.657872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:45.879119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974802+0000","last_change":"2024-03-26T21:48:30.759102+0000","last_active":"2024-03-26T21:50:09.974802+0000","last_peered":"2024-03-26T21:50:09.974802+0000","last_clean":"2024-03-26T21:50:09.974802+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:50:09.974802+0000","last_undegraded":"2024-03-26T21:50:09.974802+0000","last_fullsized":"2024-03-26T21:50:09.974802+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.758986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.758986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"480'10220","reported_seq":10325,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.296983+0000","last_change":"2024-03-26T21:47:39.053277+0000","last_active":"2024-03-26T21:50:13.296983+0000","last_peered":"2024-03-26T21:50:13.296983+0000","last_clean":"2024-03-26T21:50:13.296983+0000","last_became_active":"2024-03-26T21:47:39.052942+0000","last_became_peered":"2024-03-26T21:47:39.052942+0000","last_unstale":"2024-03-26T21:50:13.296983+0000","last_undegraded":"2024-03-26T21:50:13.296983+0000","last_fullsized":"2024-03-26T21:50:13.296983+0000","mapping_epoch":433,"log_start":"460'5852","ondisk_log_start":"460'5852","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292159488,"num_objects":4458,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10220,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1068,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972768+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:50:09.972768+0000","last_peered":"2024-03-26T21:50:09.972768+0000","last_clean":"2024-03-26T21:50:09.972768+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:50:09.972768+0000","last_undegraded":"2024-03-26T21:50:09.972768+0000","last_fullsized":"2024-03-26T21:50:09.972768+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.977514+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:50:09.977514+0000","last_peered":"2024-03-26T21:50:09.977514+0000","last_clean":"2024-03-26T21:50:09.977514+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:50:09.977514+0000","last_undegraded":"2024-03-26T21:50:09.977514+0000","last_fullsized":"2024-03-26T21:50:09.977514+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":771,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966736+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:50:09.966736+0000","last_peered":"2024-03-26T21:50:09.966736+0000","last_clean":"2024-03-26T21:50:09.966736+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:50:09.966736+0000","last_undegraded":"2024-03-26T21:50:09.966736+0000","last_fullsized":"2024-03-26T21:50:09.966736+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":904,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967691+0000","last_change":"2024-03-26T21:50:01.887630+0000","last_active":"2024-03-26T21:50:09.967691+0000","last_peered":"2024-03-26T21:50:09.967691+0000","last_clean":"2024-03-26T21:50:09.967691+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:50:09.967691+0000","last_undegraded":"2024-03-26T21:50:09.967691+0000","last_fullsized":"2024-03-26T21:50:09.967691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:01.887146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:50:01.887146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:20.427066+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"480'10190","reported_seq":23953,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.416392+0000","last_change":"2024-03-26T21:49:45.255955+0000","last_active":"2024-03-26T21:50:13.416392+0000","last_peered":"2024-03-26T21:50:13.416392+0000","last_clean":"2024-03-26T21:50:13.416392+0000","last_became_active":"2024-03-26T21:49:45.248068+0000","last_became_peered":"2024-03-26T21:49:45.248068+0000","last_unstale":"2024-03-26T21:50:13.416392+0000","last_undegraded":"2024-03-26T21:50:13.416392+0000","last_fullsized":"2024-03-26T21:50:13.416392+0000","mapping_epoch":471,"log_start":"458'5491","ondisk_log_start":"458'5491","created":433,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300154880,"num_objects":4580,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10190,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6823,"num_bytes_recovered":447152128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":618,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966307+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:50:09.966307+0000","last_peered":"2024-03-26T21:50:09.966307+0000","last_clean":"2024-03-26T21:50:09.966307+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:50:09.966307+0000","last_undegraded":"2024-03-26T21:50:09.966307+0000","last_fullsized":"2024-03-26T21:50:09.966307+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":289,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976159+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:50:09.976159+0000","last_peered":"2024-03-26T21:50:09.976159+0000","last_clean":"2024-03-26T21:50:09.976159+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:50:09.976159+0000","last_undegraded":"2024-03-26T21:50:09.976159+0000","last_fullsized":"2024-03-26T21:50:09.976159+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976329+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:50:09.976329+0000","last_peered":"2024-03-26T21:50:09.976329+0000","last_clean":"2024-03-26T21:50:09.976329+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:50:09.976329+0000","last_undegraded":"2024-03-26T21:50:09.976329+0000","last_fullsized":"2024-03-26T21:50:09.976329+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":625,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964970+0000","last_change":"2024-03-26T21:48:58.977779+0000","last_active":"2024-03-26T21:50:09.964970+0000","last_peered":"2024-03-26T21:50:09.964970+0000","last_clean":"2024-03-26T21:50:09.964970+0000","last_became_active":"2024-03-26T21:48:58.977356+0000","last_became_peered":"2024-03-26T21:48:58.977356+0000","last_unstale":"2024-03-26T21:50:09.964970+0000","last_undegraded":"2024-03-26T21:50:09.964970+0000","last_fullsized":"2024-03-26T21:50:09.964970+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:28.651656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:48:28.651656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"480'10069","reported_seq":10173,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:11.389407+0000","last_change":"2024-03-26T21:47:39.053658+0000","last_active":"2024-03-26T21:50:11.389407+0000","last_peered":"2024-03-26T21:50:11.389407+0000","last_clean":"2024-03-26T21:50:11.389407+0000","last_became_active":"2024-03-26T21:47:39.053244+0000","last_became_peered":"2024-03-26T21:47:39.053244+0000","last_unstale":"2024-03-26T21:50:11.389407+0000","last_undegraded":"2024-03-26T21:50:11.389407+0000","last_fullsized":"2024-03-26T21:50:11.389407+0000","mapping_epoch":433,"log_start":"456'5298","ondisk_log_start":"456'5298","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305594368,"num_objects":4663,"num_object_clones":0,"num_object_copies":13989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":471424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":322,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.977648+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:50:09.977648+0000","last_peered":"2024-03-26T21:50:09.977648+0000","last_clean":"2024-03-26T21:50:09.977648+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:50:09.977648+0000","last_undegraded":"2024-03-26T21:50:09.977648+0000","last_fullsized":"2024-03-26T21:50:09.977648+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":657,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965716+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:50:09.965716+0000","last_peered":"2024-03-26T21:50:09.965716+0000","last_clean":"2024-03-26T21:50:09.965716+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:50:09.965716+0000","last_undegraded":"2024-03-26T21:50:09.965716+0000","last_fullsized":"2024-03-26T21:50:09.965716+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":914,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865555+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:50:06.865555+0000","last_peered":"2024-03-26T21:50:06.865555+0000","last_clean":"2024-03-26T21:50:06.865555+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:50:06.865555+0000","last_undegraded":"2024-03-26T21:50:06.865555+0000","last_fullsized":"2024-03-26T21:50:06.865555+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":828,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972724+0000","last_change":"2024-03-26T21:48:58.986678+0000","last_active":"2024-03-26T21:50:09.972724+0000","last_peered":"2024-03-26T21:50:09.972724+0000","last_clean":"2024-03-26T21:50:09.972724+0000","last_became_active":"2024-03-26T21:48:58.985775+0000","last_became_peered":"2024-03-26T21:48:58.985775+0000","last_unstale":"2024-03-26T21:50:09.972724+0000","last_undegraded":"2024-03-26T21:50:09.972724+0000","last_fullsized":"2024-03-26T21:50:09.972724+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"480'10013","reported_seq":13382,"reported_epoch":480,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:50:11.390429+0000","last_change":"2024-03-26T21:49:36.580498+0000","last_active":"2024-03-26T21:50:11.390429+0000","last_peered":"2024-03-26T21:50:11.390429+0000","last_clean":"2024-03-26T21:48:57.955814+0000","last_became_active":"2024-03-26T21:49:00.339243+0000","last_became_peered":"2024-03-26T21:49:00.339243+0000","last_unstale":"2024-03-26T21:50:11.390429+0000","last_undegraded":"2024-03-26T21:49:36.577298+0000","last_fullsized":"2024-03-26T21:48:59.964175+0000","mapping_epoch":462,"log_start":"456'5274","ondisk_log_start":"456'5274","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":3599,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311558168,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1020,"num_objects_misplaced":3631,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10013,"num_write_kb":472513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1610,"num_bytes_recovered":105512960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":3631},{"shards":"4,6","objects":104}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1160,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865643+0000","last_change":"2024-03-26T21:48:32.957198+0000","last_active":"2024-03-26T21:50:06.865643+0000","last_peered":"2024-03-26T21:50:06.865643+0000","last_clean":"2024-03-26T21:50:06.865643+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:50:06.865643+0000","last_undegraded":"2024-03-26T21:50:06.865643+0000","last_fullsized":"2024-03-26T21:50:06.865643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:48:32.957059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.957059+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":997,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971622+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:50:09.971622+0000","last_peered":"2024-03-26T21:50:09.971622+0000","last_clean":"2024-03-26T21:50:09.971622+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:50:09.971622+0000","last_undegraded":"2024-03-26T21:50:09.971622+0000","last_fullsized":"2024-03-26T21:50:09.971622+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":935,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967915+0000","last_change":"2024-03-26T21:49:27.705867+0000","last_active":"2024-03-26T21:50:09.967915+0000","last_peered":"2024-03-26T21:50:09.967915+0000","last_clean":"2024-03-26T21:50:09.967915+0000","last_became_active":"2024-03-26T21:48:58.979760+0000","last_became_peered":"2024-03-26T21:48:58.979760+0000","last_unstale":"2024-03-26T21:50:09.967915+0000","last_undegraded":"2024-03-26T21:50:09.967915+0000","last_fullsized":"2024-03-26T21:50:09.967915+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:27.705733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:49:27.705733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:33.766431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":907,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.862569+0000","last_change":"2024-03-26T21:48:58.973098+0000","last_active":"2024-03-26T21:50:06.862569+0000","last_peered":"2024-03-26T21:50:06.862569+0000","last_clean":"2024-03-26T21:50:06.862569+0000","last_became_active":"2024-03-26T21:48:58.972760+0000","last_became_peered":"2024-03-26T21:48:58.972760+0000","last_unstale":"2024-03-26T21:50:06.862569+0000","last_undegraded":"2024-03-26T21:50:06.862569+0000","last_fullsized":"2024-03-26T21:50:06.862569+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"480'10370","reported_seq":21663,"reported_epoch":480,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T21:50:13.367735+0000","last_change":"2024-03-26T21:49:30.159143+0000","last_active":"2024-03-26T21:50:13.367735+0000","last_peered":"2024-03-26T21:50:13.367735+0000","last_clean":"2024-03-26T21:48:57.973612+0000","last_became_active":"2024-03-26T21:49:00.120902+0000","last_became_peered":"2024-03-26T21:49:00.120902+0000","last_unstale":"2024-03-26T21:50:13.367735+0000","last_undegraded":"2024-03-26T21:50:13.367735+0000","last_fullsized":"2024-03-26T21:50:13.367735+0000","mapping_epoch":462,"log_start":"460'6056","ondisk_log_start":"460'6056","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311164928,"num_objects":4748,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5589,"num_bytes_recovered":366280704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":4748}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.867337+0000","last_change":"2024-03-26T21:49:29.979212+0000","last_active":"2024-03-26T21:50:06.867337+0000","last_peered":"2024-03-26T21:50:06.867337+0000","last_clean":"2024-03-26T21:50:06.867337+0000","last_became_active":"2024-03-26T21:49:29.978839+0000","last_became_peered":"2024-03-26T21:49:29.978839+0000","last_unstale":"2024-03-26T21:50:06.867337+0000","last_undegraded":"2024-03-26T21:50:06.867337+0000","last_fullsized":"2024-03-26T21:50:06.867337+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":972,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865807+0000","last_change":"2024-03-26T21:48:35.997876+0000","last_active":"2024-03-26T21:50:06.865807+0000","last_peered":"2024-03-26T21:50:06.865807+0000","last_clean":"2024-03-26T21:50:06.865807+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:50:06.865807+0000","last_undegraded":"2024-03-26T21:50:06.865807+0000","last_fullsized":"2024-03-26T21:50:06.865807+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:48:35.997722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.997722+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":933,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974640+0000","last_change":"2024-03-26T21:48:58.981784+0000","last_active":"2024-03-26T21:50:09.974640+0000","last_peered":"2024-03-26T21:50:09.974640+0000","last_clean":"2024-03-26T21:50:09.974640+0000","last_became_active":"2024-03-26T21:48:58.981168+0000","last_became_peered":"2024-03-26T21:48:58.981168+0000","last_unstale":"2024-03-26T21:50:09.974640+0000","last_undegraded":"2024-03-26T21:50:09.974640+0000","last_fullsized":"2024-03-26T21:50:09.974640+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":913,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971084+0000","last_change":"2024-03-26T21:48:58.983664+0000","last_active":"2024-03-26T21:50:09.971084+0000","last_peered":"2024-03-26T21:50:09.971084+0000","last_clean":"2024-03-26T21:50:09.971084+0000","last_became_active":"2024-03-26T21:48:58.983312+0000","last_became_peered":"2024-03-26T21:48:58.983312+0000","last_unstale":"2024-03-26T21:50:09.971084+0000","last_undegraded":"2024-03-26T21:50:09.971084+0000","last_fullsized":"2024-03-26T21:50:09.971084+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.809302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"480'10254","reported_seq":10359,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.451948+0000","last_change":"2024-03-26T21:47:39.047854+0000","last_active":"2024-03-26T21:50:13.451948+0000","last_peered":"2024-03-26T21:50:13.451948+0000","last_clean":"2024-03-26T21:50:13.451948+0000","last_became_active":"2024-03-26T21:47:39.047314+0000","last_became_peered":"2024-03-26T21:47:39.047314+0000","last_unstale":"2024-03-26T21:50:13.451948+0000","last_undegraded":"2024-03-26T21:50:13.451948+0000","last_fullsized":"2024-03-26T21:50:13.451948+0000","mapping_epoch":433,"log_start":"458'5618","ondisk_log_start":"458'5618","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":13710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10254,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":908,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968366+0000","last_change":"2024-03-26T21:49:30.726913+0000","last_active":"2024-03-26T21:50:09.968366+0000","last_peered":"2024-03-26T21:50:09.968366+0000","last_clean":"2024-03-26T21:50:09.968366+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:50:09.968366+0000","last_undegraded":"2024-03-26T21:50:09.968366+0000","last_fullsized":"2024-03-26T21:50:09.968366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:30.726780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:30.726780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.940297+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":990,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865447+0000","last_change":"2024-03-26T21:48:59.007492+0000","last_active":"2024-03-26T21:50:06.865447+0000","last_peered":"2024-03-26T21:50:06.865447+0000","last_clean":"2024-03-26T21:50:06.865447+0000","last_became_active":"2024-03-26T21:48:58.986240+0000","last_became_peered":"2024-03-26T21:48:58.986240+0000","last_unstale":"2024-03-26T21:50:06.865447+0000","last_undegraded":"2024-03-26T21:50:06.865447+0000","last_fullsized":"2024-03-26T21:50:06.865447+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2054,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:11.639407+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:50:11.639407+0000","last_peered":"2024-03-26T21:50:11.639407+0000","last_clean":"2024-03-26T21:50:11.639407+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:50:11.639407+0000","last_undegraded":"2024-03-26T21:50:11.639407+0000","last_fullsized":"2024-03-26T21:50:11.639407+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1177,"num_read_kb":1172,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":934,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.864610+0000","last_change":"2024-03-26T21:48:41.039131+0000","last_active":"2024-03-26T21:50:06.864610+0000","last_peered":"2024-03-26T21:50:06.864610+0000","last_clean":"2024-03-26T21:50:06.864610+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:50:06.864610+0000","last_undegraded":"2024-03-26T21:50:06.864610+0000","last_fullsized":"2024-03-26T21:50:06.864610+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.035559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.035559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":919,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971686+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:50:09.971686+0000","last_peered":"2024-03-26T21:50:09.971686+0000","last_clean":"2024-03-26T21:50:09.971686+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:50:09.971686+0000","last_undegraded":"2024-03-26T21:50:09.971686+0000","last_fullsized":"2024-03-26T21:50:09.971686+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"480'10419","reported_seq":10524,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.473397+0000","last_change":"2024-03-26T21:47:39.047378+0000","last_active":"2024-03-26T21:50:13.473397+0000","last_peered":"2024-03-26T21:50:13.473397+0000","last_clean":"2024-03-26T21:50:13.473397+0000","last_became_active":"2024-03-26T21:47:39.047073+0000","last_became_peered":"2024-03-26T21:47:39.047073+0000","last_unstale":"2024-03-26T21:50:13.473397+0000","last_undegraded":"2024-03-26T21:50:13.473397+0000","last_fullsized":"2024-03-26T21:50:13.473397+0000","mapping_epoch":433,"log_start":"458'5774","ondisk_log_start":"458'5774","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305201152,"num_objects":4657,"num_object_clones":0,"num_object_copies":13971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":845,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972314+0000","last_change":"2024-03-26T21:48:58.992875+0000","last_active":"2024-03-26T21:50:09.972314+0000","last_peered":"2024-03-26T21:50:09.972314+0000","last_clean":"2024-03-26T21:50:09.972314+0000","last_became_active":"2024-03-26T21:48:58.992117+0000","last_became_peered":"2024-03-26T21:48:58.992117+0000","last_unstale":"2024-03-26T21:50:09.972314+0000","last_undegraded":"2024-03-26T21:50:09.972314+0000","last_fullsized":"2024-03-26T21:50:09.972314+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"468'55","reported_seq":1289,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972877+0000","last_change":"2024-03-26T21:49:29.985516+0000","last_active":"2024-03-26T21:50:09.972877+0000","last_peered":"2024-03-26T21:50:09.972877+0000","last_clean":"2024-03-26T21:50:09.972877+0000","last_became_active":"2024-03-26T21:49:29.985117+0000","last_became_peered":"2024-03-26T21:49:29.985117+0000","last_unstale":"2024-03-26T21:50:09.972877+0000","last_undegraded":"2024-03-26T21:50:09.972877+0000","last_fullsized":"2024-03-26T21:50:09.972877+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.981183+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:50:09.981183+0000","last_peered":"2024-03-26T21:50:09.981183+0000","last_clean":"2024-03-26T21:50:09.981183+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:50:09.981183+0000","last_undegraded":"2024-03-26T21:50:09.981183+0000","last_fullsized":"2024-03-26T21:50:09.981183+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1107,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975043+0000","last_change":"2024-03-26T21:48:27.821566+0000","last_active":"2024-03-26T21:50:09.975043+0000","last_peered":"2024-03-26T21:50:09.975043+0000","last_clean":"2024-03-26T21:50:09.975043+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:50:09.975043+0000","last_undegraded":"2024-03-26T21:50:09.975043+0000","last_fullsized":"2024-03-26T21:50:09.975043+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:48:27.821430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.821430+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":929,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968091+0000","last_change":"2024-03-26T21:49:29.780022+0000","last_active":"2024-03-26T21:50:09.968091+0000","last_peered":"2024-03-26T21:50:09.968091+0000","last_clean":"2024-03-26T21:50:09.968091+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:50:09.968091+0000","last_undegraded":"2024-03-26T21:50:09.968091+0000","last_fullsized":"2024-03-26T21:50:09.968091+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:29.779883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:29.779883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:46.086766+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"480'10486","reported_seq":10590,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.463650+0000","last_change":"2024-03-26T21:47:39.047146+0000","last_active":"2024-03-26T21:50:13.463650+0000","last_peered":"2024-03-26T21:50:13.463650+0000","last_clean":"2024-03-26T21:50:13.463650+0000","last_became_active":"2024-03-26T21:47:39.046837+0000","last_became_peered":"2024-03-26T21:47:39.046837+0000","last_unstale":"2024-03-26T21:50:13.463650+0000","last_undegraded":"2024-03-26T21:50:13.463650+0000","last_fullsized":"2024-03-26T21:50:13.463650+0000","mapping_epoch":433,"log_start":"459'5934","ondisk_log_start":"459'5934","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":524,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967976+0000","last_change":"2024-03-26T21:49:19.802374+0000","last_active":"2024-03-26T21:50:09.967976+0000","last_peered":"2024-03-26T21:50:09.967976+0000","last_clean":"2024-03-26T21:50:09.967976+0000","last_became_active":"2024-03-26T21:48:58.985930+0000","last_became_peered":"2024-03-26T21:48:58.985930+0000","last_unstale":"2024-03-26T21:50:09.967976+0000","last_undegraded":"2024-03-26T21:50:09.967976+0000","last_fullsized":"2024-03-26T21:50:09.967976+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:19.802252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:49:19.802252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:21.943907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":966,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972907+0000","last_change":"2024-03-26T21:48:35.767934+0000","last_active":"2024-03-26T21:50:09.972907+0000","last_peered":"2024-03-26T21:50:09.972907+0000","last_clean":"2024-03-26T21:50:09.972907+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:50:09.972907+0000","last_undegraded":"2024-03-26T21:50:09.972907+0000","last_fullsized":"2024-03-26T21:50:09.972907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:48:35.767034+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.767034+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":734,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976009+0000","last_change":"2024-03-26T21:48:58.978047+0000","last_active":"2024-03-26T21:50:09.976009+0000","last_peered":"2024-03-26T21:50:09.976009+0000","last_clean":"2024-03-26T21:50:09.976009+0000","last_became_active":"2024-03-26T21:48:58.976535+0000","last_became_peered":"2024-03-26T21:48:58.976535+0000","last_unstale":"2024-03-26T21:50:09.976009+0000","last_undegraded":"2024-03-26T21:50:09.976009+0000","last_fullsized":"2024-03-26T21:50:09.976009+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":933,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975707+0000","last_change":"2024-03-26T21:48:58.982624+0000","last_active":"2024-03-26T21:50:09.975707+0000","last_peered":"2024-03-26T21:50:09.975707+0000","last_clean":"2024-03-26T21:50:09.975707+0000","last_became_active":"2024-03-26T21:48:58.982052+0000","last_became_peered":"2024-03-26T21:48:58.982052+0000","last_unstale":"2024-03-26T21:50:09.975707+0000","last_undegraded":"2024-03-26T21:50:09.975707+0000","last_fullsized":"2024-03-26T21:50:09.975707+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":827,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.973465+0000","last_change":"2024-03-26T21:48:31.817091+0000","last_active":"2024-03-26T21:50:09.973465+0000","last_peered":"2024-03-26T21:50:09.973465+0000","last_clean":"2024-03-26T21:50:09.973465+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:50:09.973465+0000","last_undegraded":"2024-03-26T21:50:09.973465+0000","last_fullsized":"2024-03-26T21:50:09.973465+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:31.817002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.817002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"480'10147","reported_seq":10252,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.482048+0000","last_change":"2024-03-26T21:47:39.051677+0000","last_active":"2024-03-26T21:50:13.482048+0000","last_peered":"2024-03-26T21:50:13.482048+0000","last_clean":"2024-03-26T21:50:13.482048+0000","last_became_active":"2024-03-26T21:47:39.051279+0000","last_became_peered":"2024-03-26T21:47:39.051279+0000","last_unstale":"2024-03-26T21:50:13.482048+0000","last_undegraded":"2024-03-26T21:50:13.482048+0000","last_fullsized":"2024-03-26T21:50:13.482048+0000","mapping_epoch":433,"log_start":"458'5504","ondisk_log_start":"458'5504","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299565056,"num_objects":4571,"num_object_clones":0,"num_object_copies":13713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":470976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":914,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865706+0000","last_change":"2024-03-26T21:48:32.000228+0000","last_active":"2024-03-26T21:50:06.865706+0000","last_peered":"2024-03-26T21:50:06.865706+0000","last_clean":"2024-03-26T21:50:06.865706+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:50:06.865706+0000","last_undegraded":"2024-03-26T21:50:06.865706+0000","last_fullsized":"2024-03-26T21:50:06.865706+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.000079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"462'56","reported_seq":1297,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967438+0000","last_change":"2024-03-26T21:49:57.671639+0000","last_active":"2024-03-26T21:50:09.967438+0000","last_peered":"2024-03-26T21:50:09.967438+0000","last_clean":"2024-03-26T21:50:09.967438+0000","last_became_active":"2024-03-26T21:48:58.981458+0000","last_became_peered":"2024-03-26T21:48:58.981458+0000","last_unstale":"2024-03-26T21:50:09.967438+0000","last_undegraded":"2024-03-26T21:50:09.967438+0000","last_fullsized":"2024-03-26T21:50:09.967438+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'56","last_scrub_stamp":"2024-03-26T21:49:57.671485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:49:57.671485+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:12.579557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965081+0000","last_change":"2024-03-26T21:48:58.976086+0000","last_active":"2024-03-26T21:50:09.965081+0000","last_peered":"2024-03-26T21:50:09.965081+0000","last_clean":"2024-03-26T21:50:09.965081+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:50:09.965081+0000","last_undegraded":"2024-03-26T21:50:09.965081+0000","last_fullsized":"2024-03-26T21:50:09.965081+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":927,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967839+0000","last_change":"2024-03-26T21:48:43.783835+0000","last_active":"2024-03-26T21:50:09.967839+0000","last_peered":"2024-03-26T21:50:09.967839+0000","last_clean":"2024-03-26T21:50:09.967839+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:50:09.967839+0000","last_undegraded":"2024-03-26T21:50:09.967839+0000","last_fullsized":"2024-03-26T21:50:09.967839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:48:43.783585+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976655+0000","last_change":"2024-03-26T21:48:29.771118+0000","last_active":"2024-03-26T21:50:09.976655+0000","last_peered":"2024-03-26T21:50:09.976655+0000","last_clean":"2024-03-26T21:50:09.976655+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:50:09.976655+0000","last_undegraded":"2024-03-26T21:50:09.976655+0000","last_fullsized":"2024-03-26T21:50:09.976655+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.770950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.770950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"480'10384","reported_seq":23827,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.454638+0000","last_change":"2024-03-26T21:49:57.447616+0000","last_active":"2024-03-26T21:50:13.454638+0000","last_peered":"2024-03-26T21:50:13.454638+0000","last_clean":"2024-03-26T21:50:13.454638+0000","last_became_active":"2024-03-26T21:49:57.435919+0000","last_became_peered":"2024-03-26T21:49:57.435919+0000","last_unstale":"2024-03-26T21:50:13.454638+0000","last_undegraded":"2024-03-26T21:50:13.454638+0000","last_fullsized":"2024-03-26T21:50:13.454638+0000","mapping_epoch":476,"log_start":"458'5724","ondisk_log_start":"458'5724","created":433,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301334528,"num_objects":4598,"num_object_clones":0,"num_object_copies":13794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10384,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6662,"num_bytes_recovered":436600832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":955,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865910+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:50:06.865910+0000","last_peered":"2024-03-26T21:50:06.865910+0000","last_clean":"2024-03-26T21:50:06.865910+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:50:06.865910+0000","last_undegraded":"2024-03-26T21:50:06.865910+0000","last_fullsized":"2024-03-26T21:50:06.865910+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":936,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967697+0000","last_change":"2024-03-26T21:49:31.653285+0000","last_active":"2024-03-26T21:50:09.967697+0000","last_peered":"2024-03-26T21:50:09.967697+0000","last_clean":"2024-03-26T21:50:09.967697+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:50:09.967697+0000","last_undegraded":"2024-03-26T21:50:09.967697+0000","last_fullsized":"2024-03-26T21:50:09.967697+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:31.653167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:49:31.653167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:59.137490+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":732,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964657+0000","last_change":"2024-03-26T21:48:58.968424+0000","last_active":"2024-03-26T21:50:09.964657+0000","last_peered":"2024-03-26T21:50:09.964657+0000","last_clean":"2024-03-26T21:50:09.964657+0000","last_became_active":"2024-03-26T21:48:58.968010+0000","last_became_peered":"2024-03-26T21:48:58.968010+0000","last_unstale":"2024-03-26T21:50:09.964657+0000","last_undegraded":"2024-03-26T21:50:09.964657+0000","last_fullsized":"2024-03-26T21:50:09.964657+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":895,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865100+0000","last_change":"2024-03-26T21:48:58.983049+0000","last_active":"2024-03-26T21:50:06.865100+0000","last_peered":"2024-03-26T21:50:06.865100+0000","last_clean":"2024-03-26T21:50:06.865100+0000","last_became_active":"2024-03-26T21:48:58.982744+0000","last_became_peered":"2024-03-26T21:48:58.982744+0000","last_unstale":"2024-03-26T21:50:06.865100+0000","last_undegraded":"2024-03-26T21:50:06.865100+0000","last_fullsized":"2024-03-26T21:50:06.865100+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"480'9969","reported_seq":10074,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:13.362641+0000","last_change":"2024-03-26T21:47:39.046775+0000","last_active":"2024-03-26T21:50:13.362641+0000","last_peered":"2024-03-26T21:50:13.362641+0000","last_clean":"2024-03-26T21:50:13.362641+0000","last_became_active":"2024-03-26T21:47:39.046369+0000","last_became_peered":"2024-03-26T21:47:39.046369+0000","last_unstale":"2024-03-26T21:50:13.362641+0000","last_undegraded":"2024-03-26T21:50:13.362641+0000","last_fullsized":"2024-03-26T21:50:13.362641+0000","mapping_epoch":433,"log_start":"457'5257","ondisk_log_start":"457'5257","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296550400,"num_objects":4525,"num_object_clones":0,"num_object_copies":13575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9969,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":997,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971449+0000","last_change":"2024-03-26T21:49:29.981307+0000","last_active":"2024-03-26T21:50:09.971449+0000","last_peered":"2024-03-26T21:50:09.971449+0000","last_clean":"2024-03-26T21:50:09.971449+0000","last_became_active":"2024-03-26T21:49:29.980914+0000","last_became_peered":"2024-03-26T21:49:29.980914+0000","last_unstale":"2024-03-26T21:50:09.971449+0000","last_undegraded":"2024-03-26T21:50:09.971449+0000","last_fullsized":"2024-03-26T21:50:09.971449+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":861,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975137+0000","last_change":"2024-03-26T21:48:44.612681+0000","last_active":"2024-03-26T21:50:09.975137+0000","last_peered":"2024-03-26T21:50:09.975137+0000","last_clean":"2024-03-26T21:50:09.975137+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:50:09.975137+0000","last_undegraded":"2024-03-26T21:50:09.975137+0000","last_fullsized":"2024-03-26T21:50:09.975137+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.612470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865398+0000","last_change":"2024-03-26T21:48:58.986135+0000","last_active":"2024-03-26T21:50:06.865398+0000","last_peered":"2024-03-26T21:50:06.865398+0000","last_clean":"2024-03-26T21:50:06.865398+0000","last_became_active":"2024-03-26T21:48:58.985788+0000","last_became_peered":"2024-03-26T21:48:58.985788+0000","last_unstale":"2024-03-26T21:50:06.865398+0000","last_undegraded":"2024-03-26T21:50:06.865398+0000","last_fullsized":"2024-03-26T21:50:06.865398+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":833,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.969934+0000","last_change":"2024-03-26T21:48:58.984676+0000","last_active":"2024-03-26T21:50:09.969934+0000","last_peered":"2024-03-26T21:50:09.969934+0000","last_clean":"2024-03-26T21:50:09.969934+0000","last_became_active":"2024-03-26T21:48:58.983980+0000","last_became_peered":"2024-03-26T21:48:58.983980+0000","last_unstale":"2024-03-26T21:50:09.969934+0000","last_undegraded":"2024-03-26T21:50:09.969934+0000","last_fullsized":"2024-03-26T21:50:09.969934+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"479'9831","reported_seq":23454,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.071576+0000","last_change":"2024-03-26T21:49:45.257806+0000","last_active":"2024-03-26T21:50:09.071576+0000","last_peered":"2024-03-26T21:50:09.071576+0000","last_clean":"2024-03-26T21:50:09.071576+0000","last_became_active":"2024-03-26T21:49:45.247163+0000","last_became_peered":"2024-03-26T21:49:45.247163+0000","last_unstale":"2024-03-26T21:50:09.071576+0000","last_undegraded":"2024-03-26T21:50:09.071576+0000","last_fullsized":"2024-03-26T21:50:09.071576+0000","mapping_epoch":471,"log_start":"454'5011","ondisk_log_start":"454'5011","created":433,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318439424,"num_objects":4859,"num_object_clones":0,"num_object_copies":14577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6755,"num_bytes_recovered":442695680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.862842+0000","last_change":"2024-03-26T21:48:58.977207+0000","last_active":"2024-03-26T21:50:06.862842+0000","last_peered":"2024-03-26T21:50:06.862842+0000","last_clean":"2024-03-26T21:50:06.862842+0000","last_became_active":"2024-03-26T21:48:58.976826+0000","last_became_peered":"2024-03-26T21:48:58.976826+0000","last_unstale":"2024-03-26T21:50:06.862842+0000","last_undegraded":"2024-03-26T21:50:06.862842+0000","last_fullsized":"2024-03-26T21:50:06.862842+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":670,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966155+0000","last_change":"2024-03-26T21:48:36.706426+0000","last_active":"2024-03-26T21:50:09.966155+0000","last_peered":"2024-03-26T21:50:09.966155+0000","last_clean":"2024-03-26T21:50:09.966155+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:50:09.966155+0000","last_undegraded":"2024-03-26T21:50:09.966155+0000","last_fullsized":"2024-03-26T21:50:09.966155+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:48:36.706278+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":891,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971683+0000","last_change":"2024-03-26T21:48:35.727970+0000","last_active":"2024-03-26T21:50:09.971683+0000","last_peered":"2024-03-26T21:50:09.971683+0000","last_clean":"2024-03-26T21:50:09.971683+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:50:09.971683+0000","last_undegraded":"2024-03-26T21:50:09.971683+0000","last_fullsized":"2024-03-26T21:50:09.971683+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.727841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975353+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:50:09.975353+0000","last_peered":"2024-03-26T21:50:09.975353+0000","last_clean":"2024-03-26T21:50:09.975353+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:50:09.975353+0000","last_undegraded":"2024-03-26T21:50:09.975353+0000","last_fullsized":"2024-03-26T21:50:09.975353+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"480'10207","reported_seq":10393,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:12.145795+0000","last_change":"2024-03-26T21:49:25.777937+0000","last_active":"2024-03-26T21:50:12.145795+0000","last_peered":"2024-03-26T21:50:12.145795+0000","last_clean":"2024-03-26T21:50:12.145795+0000","last_became_active":"2024-03-26T21:47:39.051349+0000","last_became_peered":"2024-03-26T21:47:39.051349+0000","last_unstale":"2024-03-26T21:50:12.145795+0000","last_undegraded":"2024-03-26T21:50:12.145795+0000","last_fullsized":"2024-03-26T21:50:12.145795+0000","mapping_epoch":433,"log_start":"454'4937","ondisk_log_start":"454'4937","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'7654","last_scrub_stamp":"2024-03-26T21:49:25.777551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:49:25.777551+0000","objects_scrubbed":7294,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:47.024151+0000","scrub_duration":2010,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306774016,"num_objects":4681,"num_object_clones":0,"num_object_copies":14043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10207,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1107,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.970320+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:50:09.970320+0000","last_peered":"2024-03-26T21:50:09.970320+0000","last_clean":"2024-03-26T21:50:09.970320+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:50:09.970320+0000","last_undegraded":"2024-03-26T21:50:09.970320+0000","last_fullsized":"2024-03-26T21:50:09.970320+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":989,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967184+0000","last_change":"2024-03-26T21:49:38.645618+0000","last_active":"2024-03-26T21:50:09.967184+0000","last_peered":"2024-03-26T21:50:09.967184+0000","last_clean":"2024-03-26T21:50:09.967184+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:50:09.967184+0000","last_undegraded":"2024-03-26T21:50:09.967184+0000","last_fullsized":"2024-03-26T21:50:09.967184+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:49:38.645456+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.832210+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972194+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:50:09.972194+0000","last_peered":"2024-03-26T21:50:09.972194+0000","last_clean":"2024-03-26T21:50:09.972194+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:50:09.972194+0000","last_undegraded":"2024-03-26T21:50:09.972194+0000","last_fullsized":"2024-03-26T21:50:09.972194+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":904,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967991+0000","last_change":"2024-03-26T21:49:35.727593+0000","last_active":"2024-03-26T21:50:09.967991+0000","last_peered":"2024-03-26T21:50:09.967991+0000","last_clean":"2024-03-26T21:50:09.967991+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:50:09.967991+0000","last_undegraded":"2024-03-26T21:50:09.967991+0000","last_fullsized":"2024-03-26T21:50:09.967991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:35.727500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:35.727500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:52.939494+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":284,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976017+0000","last_change":"2024-03-26T21:48:59.002050+0000","last_active":"2024-03-26T21:50:09.976017+0000","last_peered":"2024-03-26T21:50:09.976017+0000","last_clean":"2024-03-26T21:50:09.976017+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:50:09.976017+0000","last_undegraded":"2024-03-26T21:50:09.976017+0000","last_fullsized":"2024-03-26T21:50:09.976017+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.862935+0000","last_change":"2024-03-26T21:49:29.977391+0000","last_active":"2024-03-26T21:50:06.862935+0000","last_peered":"2024-03-26T21:50:06.862935+0000","last_clean":"2024-03-26T21:50:06.862935+0000","last_became_active":"2024-03-26T21:49:29.976836+0000","last_became_peered":"2024-03-26T21:49:29.976836+0000","last_unstale":"2024-03-26T21:50:06.862935+0000","last_undegraded":"2024-03-26T21:50:06.862935+0000","last_fullsized":"2024-03-26T21:50:06.862935+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.291965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":919,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968148+0000","last_change":"2024-03-26T21:49:41.705007+0000","last_active":"2024-03-26T21:50:09.968148+0000","last_peered":"2024-03-26T21:50:09.968148+0000","last_clean":"2024-03-26T21:50:09.968148+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:50:09.968148+0000","last_undegraded":"2024-03-26T21:50:09.968148+0000","last_fullsized":"2024-03-26T21:50:09.968148+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:41.704871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:41.704871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:49.588465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":836,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.973560+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:50:09.973560+0000","last_peered":"2024-03-26T21:50:09.973560+0000","last_clean":"2024-03-26T21:50:09.973560+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:50:09.973560+0000","last_undegraded":"2024-03-26T21:50:09.973560+0000","last_fullsized":"2024-03-26T21:50:09.973560+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":972,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865772+0000","last_change":"2024-03-26T21:48:44.101005+0000","last_active":"2024-03-26T21:50:06.865772+0000","last_peered":"2024-03-26T21:50:06.865772+0000","last_clean":"2024-03-26T21:50:06.865772+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:50:06.865772+0000","last_undegraded":"2024-03-26T21:50:06.865772+0000","last_fullsized":"2024-03-26T21:50:06.865772+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:48:44.100867+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.100867+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:07.849510+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":873,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.970402+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:50:09.970402+0000","last_peered":"2024-03-26T21:50:09.970402+0000","last_clean":"2024-03-26T21:50:09.970402+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:50:09.970402+0000","last_undegraded":"2024-03-26T21:50:09.970402+0000","last_fullsized":"2024-03-26T21:50:09.970402+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976184+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:50:09.976184+0000","last_peered":"2024-03-26T21:50:09.976184+0000","last_clean":"2024-03-26T21:50:09.976184+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:50:09.976184+0000","last_undegraded":"2024-03-26T21:50:09.976184+0000","last_fullsized":"2024-03-26T21:50:09.976184+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":533,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965353+0000","last_change":"2024-03-26T21:48:58.984295+0000","last_active":"2024-03-26T21:50:09.965353+0000","last_peered":"2024-03-26T21:50:09.965353+0000","last_clean":"2024-03-26T21:50:09.965353+0000","last_became_active":"2024-03-26T21:48:58.983139+0000","last_became_peered":"2024-03-26T21:48:58.983139+0000","last_unstale":"2024-03-26T21:50:09.965353+0000","last_undegraded":"2024-03-26T21:50:09.965353+0000","last_fullsized":"2024-03-26T21:50:09.965353+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":988,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968079+0000","last_change":"2024-03-26T21:50:00.738089+0000","last_active":"2024-03-26T21:50:09.968079+0000","last_peered":"2024-03-26T21:50:09.968079+0000","last_clean":"2024-03-26T21:50:09.968079+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:50:09.968079+0000","last_undegraded":"2024-03-26T21:50:09.968079+0000","last_fullsized":"2024-03-26T21:50:09.968079+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:50:00.737954+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:50:00.737954+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:01.952910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":911,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.867250+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:50:06.867250+0000","last_peered":"2024-03-26T21:50:06.867250+0000","last_clean":"2024-03-26T21:50:06.867250+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:50:06.867250+0000","last_undegraded":"2024-03-26T21:50:06.867250+0000","last_fullsized":"2024-03-26T21:50:06.867250+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"462'14","reported_seq":333,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.978243+0000","last_change":"2024-03-26T21:48:58.993774+0000","last_active":"2024-03-26T21:50:09.978243+0000","last_peered":"2024-03-26T21:50:09.978243+0000","last_clean":"2024-03-26T21:50:09.978243+0000","last_became_active":"2024-03-26T21:48:58.989325+0000","last_became_peered":"2024-03-26T21:48:58.989325+0000","last_unstale":"2024-03-26T21:50:09.978243+0000","last_undegraded":"2024-03-26T21:50:09.978243+0000","last_fullsized":"2024-03-26T21:50:09.978243+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13","last_scrub_stamp":"2024-03-26T21:48:33.859117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.859117+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":603,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966799+0000","last_change":"2024-03-26T21:48:58.970848+0000","last_active":"2024-03-26T21:50:09.966799+0000","last_peered":"2024-03-26T21:50:09.966799+0000","last_clean":"2024-03-26T21:50:09.966799+0000","last_became_active":"2024-03-26T21:48:58.970509+0000","last_became_peered":"2024-03-26T21:48:58.970509+0000","last_unstale":"2024-03-26T21:50:09.966799+0000","last_undegraded":"2024-03-26T21:50:09.966799+0000","last_fullsized":"2024-03-26T21:50:09.966799+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":280,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974694+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:50:09.974694+0000","last_peered":"2024-03-26T21:50:09.974694+0000","last_clean":"2024-03-26T21:50:09.974694+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:50:09.974694+0000","last_undegraded":"2024-03-26T21:50:09.974694+0000","last_fullsized":"2024-03-26T21:50:09.974694+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":852,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972877+0000","last_change":"2024-03-26T21:48:58.987967+0000","last_active":"2024-03-26T21:50:09.972877+0000","last_peered":"2024-03-26T21:50:09.972877+0000","last_clean":"2024-03-26T21:50:09.972877+0000","last_became_active":"2024-03-26T21:48:58.987113+0000","last_became_peered":"2024-03-26T21:48:58.987113+0000","last_unstale":"2024-03-26T21:50:09.972877+0000","last_undegraded":"2024-03-26T21:50:09.972877+0000","last_fullsized":"2024-03-26T21:50:09.972877+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1176,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968095+0000","last_change":"2024-03-26T21:50:05.674011+0000","last_active":"2024-03-26T21:50:09.968095+0000","last_peered":"2024-03-26T21:50:09.968095+0000","last_clean":"2024-03-26T21:50:09.968095+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:50:09.968095+0000","last_undegraded":"2024-03-26T21:50:09.968095+0000","last_fullsized":"2024-03-26T21:50:09.968095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:50:05.673903+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:50:05.673903+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:21.698146+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":907,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865337+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:50:06.865337+0000","last_peered":"2024-03-26T21:50:06.865337+0000","last_clean":"2024-03-26T21:50:06.865337+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:50:06.865337+0000","last_undegraded":"2024-03-26T21:50:06.865337+0000","last_fullsized":"2024-03-26T21:50:06.865337+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":639,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965981+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:50:09.965981+0000","last_peered":"2024-03-26T21:50:09.965981+0000","last_clean":"2024-03-26T21:50:09.965981+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:50:09.965981+0000","last_undegraded":"2024-03-26T21:50:09.965981+0000","last_fullsized":"2024-03-26T21:50:09.965981+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":607,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965643+0000","last_change":"2024-03-26T21:48:39.735268+0000","last_active":"2024-03-26T21:50:09.965643+0000","last_peered":"2024-03-26T21:50:09.965643+0000","last_clean":"2024-03-26T21:50:09.965643+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:50:09.965643+0000","last_undegraded":"2024-03-26T21:50:09.965643+0000","last_fullsized":"2024-03-26T21:50:09.965643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:48:39.735146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":934,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971539+0000","last_change":"2024-03-26T21:48:24.700288+0000","last_active":"2024-03-26T21:50:09.971539+0000","last_peered":"2024-03-26T21:50:09.971539+0000","last_clean":"2024-03-26T21:50:09.971539+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:50:09.971539+0000","last_undegraded":"2024-03-26T21:50:09.971539+0000","last_fullsized":"2024-03-26T21:50:09.971539+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:48:24.700160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976315+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:50:09.976315+0000","last_peered":"2024-03-26T21:50:09.976315+0000","last_clean":"2024-03-26T21:50:09.976315+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:50:09.976315+0000","last_undegraded":"2024-03-26T21:50:09.976315+0000","last_fullsized":"2024-03-26T21:50:09.976315+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":968,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971715+0000","last_change":"2024-03-26T21:48:59.005271+0000","last_active":"2024-03-26T21:50:09.971715+0000","last_peered":"2024-03-26T21:50:09.971715+0000","last_clean":"2024-03-26T21:50:09.971715+0000","last_became_active":"2024-03-26T21:48:58.979497+0000","last_became_peered":"2024-03-26T21:48:58.979497+0000","last_unstale":"2024-03-26T21:50:09.971715+0000","last_undegraded":"2024-03-26T21:50:09.971715+0000","last_fullsized":"2024-03-26T21:50:09.971715+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:30.679017+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.679017+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":851,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.973652+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:50:09.973652+0000","last_peered":"2024-03-26T21:50:09.973652+0000","last_clean":"2024-03-26T21:50:09.973652+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:50:09.973652+0000","last_undegraded":"2024-03-26T21:50:09.973652+0000","last_fullsized":"2024-03-26T21:50:09.973652+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975191+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:50:09.975191+0000","last_peered":"2024-03-26T21:50:09.975191+0000","last_clean":"2024-03-26T21:50:09.975191+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:50:09.975191+0000","last_undegraded":"2024-03-26T21:50:09.975191+0000","last_fullsized":"2024-03-26T21:50:09.975191+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":888,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.862298+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:50:06.862298+0000","last_peered":"2024-03-26T21:50:06.862298+0000","last_clean":"2024-03-26T21:50:06.862298+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:50:06.862298+0000","last_undegraded":"2024-03-26T21:50:06.862298+0000","last_fullsized":"2024-03-26T21:50:06.862298+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":285,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.977274+0000","last_change":"2024-03-26T21:48:38.975837+0000","last_active":"2024-03-26T21:50:09.977274+0000","last_peered":"2024-03-26T21:50:09.977274+0000","last_clean":"2024-03-26T21:50:09.977274+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:50:09.977274+0000","last_undegraded":"2024-03-26T21:50:09.977274+0000","last_fullsized":"2024-03-26T21:50:09.977274+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:38.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:38.975710+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":924,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967686+0000","last_change":"2024-03-26T21:49:55.627083+0000","last_active":"2024-03-26T21:50:09.967686+0000","last_peered":"2024-03-26T21:50:09.967686+0000","last_clean":"2024-03-26T21:50:09.967686+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:50:09.967686+0000","last_undegraded":"2024-03-26T21:50:09.967686+0000","last_fullsized":"2024-03-26T21:50:09.967686+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:55.626947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:49:55.626947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:07.640931+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":828,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974939+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:50:09.974939+0000","last_peered":"2024-03-26T21:50:09.974939+0000","last_clean":"2024-03-26T21:50:09.974939+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:50:09.974939+0000","last_undegraded":"2024-03-26T21:50:09.974939+0000","last_fullsized":"2024-03-26T21:50:09.974939+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":261,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975271+0000","last_change":"2024-03-26T21:48:37.104233+0000","last_active":"2024-03-26T21:50:09.975271+0000","last_peered":"2024-03-26T21:50:09.975271+0000","last_clean":"2024-03-26T21:50:09.975271+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:50:09.975271+0000","last_undegraded":"2024-03-26T21:50:09.975271+0000","last_fullsized":"2024-03-26T21:50:09.975271+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.104108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.104108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":273,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976452+0000","last_change":"2024-03-26T21:48:42.913651+0000","last_active":"2024-03-26T21:50:09.976452+0000","last_peered":"2024-03-26T21:50:09.976452+0000","last_clean":"2024-03-26T21:50:09.976452+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:50:09.976452+0000","last_undegraded":"2024-03-26T21:50:09.976452+0000","last_fullsized":"2024-03-26T21:50:09.976452+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:48:42.913438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:42.913438+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":911,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.866040+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:50:06.866040+0000","last_peered":"2024-03-26T21:50:06.866040+0000","last_clean":"2024-03-26T21:50:06.866040+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:50:06.866040+0000","last_undegraded":"2024-03-26T21:50:06.866040+0000","last_fullsized":"2024-03-26T21:50:06.866040+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":525,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964361+0000","last_change":"2024-03-26T21:49:29.976926+0000","last_active":"2024-03-26T21:50:09.964361+0000","last_peered":"2024-03-26T21:50:09.964361+0000","last_clean":"2024-03-26T21:50:09.964361+0000","last_became_active":"2024-03-26T21:49:29.976388+0000","last_became_peered":"2024-03-26T21:49:29.976388+0000","last_unstale":"2024-03-26T21:50:09.964361+0000","last_undegraded":"2024-03-26T21:50:09.964361+0000","last_fullsized":"2024-03-26T21:50:09.964361+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":842,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.970182+0000","last_change":"2024-03-26T21:48:58.980520+0000","last_active":"2024-03-26T21:50:09.970182+0000","last_peered":"2024-03-26T21:50:09.970182+0000","last_clean":"2024-03-26T21:50:09.970182+0000","last_became_active":"2024-03-26T21:48:58.980151+0000","last_became_peered":"2024-03-26T21:48:58.980151+0000","last_unstale":"2024-03-26T21:50:09.970182+0000","last_undegraded":"2024-03-26T21:50:09.970182+0000","last_fullsized":"2024-03-26T21:50:09.970182+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":964,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.968273+0000","last_change":"2024-03-26T21:49:50.705427+0000","last_active":"2024-03-26T21:50:09.968273+0000","last_peered":"2024-03-26T21:50:09.968273+0000","last_clean":"2024-03-26T21:50:09.968273+0000","last_became_active":"2024-03-26T21:49:29.980233+0000","last_became_peered":"2024-03-26T21:49:29.980233+0000","last_unstale":"2024-03-26T21:50:09.968273+0000","last_undegraded":"2024-03-26T21:50:09.968273+0000","last_fullsized":"2024-03-26T21:50:09.968273+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:49:50.705277+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:49:50.705277+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:06.503329+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976053+0000","last_change":"2024-03-26T21:48:26.761549+0000","last_active":"2024-03-26T21:50:09.976053+0000","last_peered":"2024-03-26T21:50:09.976053+0000","last_clean":"2024-03-26T21:50:09.976053+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:50:09.976053+0000","last_undegraded":"2024-03-26T21:50:09.976053+0000","last_fullsized":"2024-03-26T21:50:09.976053+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.760812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.760812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":908,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865963+0000","last_change":"2024-03-26T21:48:58.986355+0000","last_active":"2024-03-26T21:50:06.865963+0000","last_peered":"2024-03-26T21:50:06.865963+0000","last_clean":"2024-03-26T21:50:06.865963+0000","last_became_active":"2024-03-26T21:48:58.985913+0000","last_became_peered":"2024-03-26T21:48:58.985913+0000","last_unstale":"2024-03-26T21:50:06.865963+0000","last_undegraded":"2024-03-26T21:50:06.865963+0000","last_fullsized":"2024-03-26T21:50:06.865963+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:48:45.073486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":926,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.983794+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:50:09.983794+0000","last_peered":"2024-03-26T21:50:09.983794+0000","last_clean":"2024-03-26T21:50:09.983794+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:50:09.983794+0000","last_undegraded":"2024-03-26T21:50:09.983794+0000","last_fullsized":"2024-03-26T21:50:09.983794+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":995,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865660+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:50:06.865660+0000","last_peered":"2024-03-26T21:50:06.865660+0000","last_clean":"2024-03-26T21:50:06.865660+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:50:06.865660+0000","last_undegraded":"2024-03-26T21:50:06.865660+0000","last_fullsized":"2024-03-26T21:50:06.865660+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":960,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.971258+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:50:09.971258+0000","last_peered":"2024-03-26T21:50:09.971258+0000","last_clean":"2024-03-26T21:50:09.971258+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:50:09.971258+0000","last_undegraded":"2024-03-26T21:50:09.971258+0000","last_fullsized":"2024-03-26T21:50:09.971258+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"468'15","reported_seq":489,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966675+0000","last_change":"2024-03-26T21:49:29.976278+0000","last_active":"2024-03-26T21:50:09.966675+0000","last_peered":"2024-03-26T21:50:09.966675+0000","last_clean":"2024-03-26T21:50:09.966675+0000","last_became_active":"2024-03-26T21:49:29.975921+0000","last_became_peered":"2024-03-26T21:49:29.975921+0000","last_unstale":"2024-03-26T21:50:09.966675+0000","last_undegraded":"2024-03-26T21:50:09.966675+0000","last_fullsized":"2024-03-26T21:50:09.966675+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:48:32.327875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.327875+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":842,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.972158+0000","last_change":"2024-03-26T21:48:58.998136+0000","last_active":"2024-03-26T21:50:09.972158+0000","last_peered":"2024-03-26T21:50:09.972158+0000","last_clean":"2024-03-26T21:50:09.972158+0000","last_became_active":"2024-03-26T21:48:58.991425+0000","last_became_peered":"2024-03-26T21:48:58.991425+0000","last_unstale":"2024-03-26T21:50:09.972158+0000","last_undegraded":"2024-03-26T21:50:09.972158+0000","last_fullsized":"2024-03-26T21:50:09.972158+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":279,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976695+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:50:09.976695+0000","last_peered":"2024-03-26T21:50:09.976695+0000","last_clean":"2024-03-26T21:50:09.976695+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:50:09.976695+0000","last_undegraded":"2024-03-26T21:50:09.976695+0000","last_fullsized":"2024-03-26T21:50:09.976695+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":604,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964885+0000","last_change":"2024-03-26T21:48:58.973845+0000","last_active":"2024-03-26T21:50:09.964885+0000","last_peered":"2024-03-26T21:50:09.964885+0000","last_clean":"2024-03-26T21:50:09.964885+0000","last_became_active":"2024-03-26T21:48:58.973428+0000","last_became_peered":"2024-03-26T21:48:58.973428+0000","last_unstale":"2024-03-26T21:50:09.964885+0000","last_undegraded":"2024-03-26T21:50:09.964885+0000","last_fullsized":"2024-03-26T21:50:09.964885+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":628,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965865+0000","last_change":"2024-03-26T21:48:58.983537+0000","last_active":"2024-03-26T21:50:09.965865+0000","last_peered":"2024-03-26T21:50:09.965865+0000","last_clean":"2024-03-26T21:50:09.965865+0000","last_became_active":"2024-03-26T21:48:58.982850+0000","last_became_peered":"2024-03-26T21:48:58.982850+0000","last_unstale":"2024-03-26T21:50:09.965865+0000","last_undegraded":"2024-03-26T21:50:09.965865+0000","last_fullsized":"2024-03-26T21:50:09.965865+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976281+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:50:09.976281+0000","last_peered":"2024-03-26T21:50:09.976281+0000","last_clean":"2024-03-26T21:50:09.976281+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:50:09.976281+0000","last_undegraded":"2024-03-26T21:50:09.976281+0000","last_fullsized":"2024-03-26T21:50:09.976281+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1040,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.974523+0000","last_change":"2024-03-26T21:48:59.014048+0000","last_active":"2024-03-26T21:50:09.974523+0000","last_peered":"2024-03-26T21:50:09.974523+0000","last_clean":"2024-03-26T21:50:09.974523+0000","last_became_active":"2024-03-26T21:48:58.977341+0000","last_became_peered":"2024-03-26T21:48:58.977341+0000","last_unstale":"2024-03-26T21:50:09.974523+0000","last_undegraded":"2024-03-26T21:50:09.974523+0000","last_fullsized":"2024-03-26T21:50:09.974523+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":921,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865783+0000","last_change":"2024-03-26T21:48:40.024866+0000","last_active":"2024-03-26T21:50:06.865783+0000","last_peered":"2024-03-26T21:50:06.865783+0000","last_clean":"2024-03-26T21:50:06.865783+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:50:06.865783+0000","last_undegraded":"2024-03-26T21:50:06.865783+0000","last_fullsized":"2024-03-26T21:50:06.865783+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.024681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.024681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":922,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.984069+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:50:09.984069+0000","last_peered":"2024-03-26T21:50:09.984069+0000","last_clean":"2024-03-26T21:50:09.984069+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:50:09.984069+0000","last_undegraded":"2024-03-26T21:50:09.984069+0000","last_fullsized":"2024-03-26T21:50:09.984069+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":599,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966724+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:50:09.966724+0000","last_peered":"2024-03-26T21:50:09.966724+0000","last_clean":"2024-03-26T21:50:09.966724+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:50:09.966724+0000","last_undegraded":"2024-03-26T21:50:09.966724+0000","last_fullsized":"2024-03-26T21:50:09.966724+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":634,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.966089+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:50:09.966089+0000","last_peered":"2024-03-26T21:50:09.966089+0000","last_clean":"2024-03-26T21:50:09.966089+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:50:09.966089+0000","last_undegraded":"2024-03-26T21:50:09.966089+0000","last_fullsized":"2024-03-26T21:50:09.966089+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":936,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967859+0000","last_change":"2024-03-26T21:50:09.758825+0000","last_active":"2024-03-26T21:50:09.967859+0000","last_peered":"2024-03-26T21:50:09.967859+0000","last_clean":"2024-03-26T21:50:09.967859+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:50:09.967859+0000","last_undegraded":"2024-03-26T21:50:09.967859+0000","last_fullsized":"2024-03-26T21:50:09.967859+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:09.758697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:50:09.758697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:13.447825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":568,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.970640+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:50:09.970640+0000","last_peered":"2024-03-26T21:50:09.970640+0000","last_clean":"2024-03-26T21:50:09.970640+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:50:09.970640+0000","last_undegraded":"2024-03-26T21:50:09.970640+0000","last_fullsized":"2024-03-26T21:50:09.970640+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":915,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967364+0000","last_change":"2024-03-26T21:49:33.717279+0000","last_active":"2024-03-26T21:50:09.967364+0000","last_peered":"2024-03-26T21:50:09.967364+0000","last_clean":"2024-03-26T21:50:09.967364+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:50:09.967364+0000","last_undegraded":"2024-03-26T21:50:09.967364+0000","last_fullsized":"2024-03-26T21:50:09.967364+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:33.716995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:49:33.716995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:48.354109+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":666,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.965004+0000","last_change":"2024-03-26T21:48:27.673802+0000","last_active":"2024-03-26T21:50:09.965004+0000","last_peered":"2024-03-26T21:50:09.965004+0000","last_clean":"2024-03-26T21:50:09.965004+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:50:09.965004+0000","last_undegraded":"2024-03-26T21:50:09.965004+0000","last_fullsized":"2024-03-26T21:50:09.965004+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:27.673516+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.673516+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":936,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967842+0000","last_change":"2024-03-26T21:49:48.724032+0000","last_active":"2024-03-26T21:50:09.967842+0000","last_peered":"2024-03-26T21:50:09.967842+0000","last_clean":"2024-03-26T21:50:09.967842+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:50:09.967842+0000","last_undegraded":"2024-03-26T21:50:09.967842+0000","last_fullsized":"2024-03-26T21:50:09.967842+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:48.723773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:49:48.723773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:59.774563+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.967396+0000","last_change":"2024-03-26T21:50:06.681825+0000","last_active":"2024-03-26T21:50:09.967396+0000","last_peered":"2024-03-26T21:50:09.967396+0000","last_clean":"2024-03-26T21:50:09.967396+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:50:09.967396+0000","last_undegraded":"2024-03-26T21:50:09.967396+0000","last_fullsized":"2024-03-26T21:50:09.967396+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_clean_scrub_stamp":"2024-03-26T21:50:06.681696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:09.917933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":888,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.862757+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:50:06.862757+0000","last_peered":"2024-03-26T21:50:06.862757+0000","last_clean":"2024-03-26T21:50:06.862757+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:50:06.862757+0000","last_undegraded":"2024-03-26T21:50:06.862757+0000","last_fullsized":"2024-03-26T21:50:06.862757+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":309,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976234+0000","last_change":"2024-03-26T21:48:59.006965+0000","last_active":"2024-03-26T21:50:09.976234+0000","last_peered":"2024-03-26T21:50:09.976234+0000","last_clean":"2024-03-26T21:50:09.976234+0000","last_became_active":"2024-03-26T21:48:58.977997+0000","last_became_peered":"2024-03-26T21:48:58.977997+0000","last_unstale":"2024-03-26T21:50:09.976234+0000","last_undegraded":"2024-03-26T21:50:09.976234+0000","last_fullsized":"2024-03-26T21:50:09.976234+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:34.910272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.910272+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":259,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.976614+0000","last_change":"2024-03-26T21:48:58.976105+0000","last_active":"2024-03-26T21:50:09.976614+0000","last_peered":"2024-03-26T21:50:09.976614+0000","last_clean":"2024-03-26T21:50:09.976614+0000","last_became_active":"2024-03-26T21:48:58.975714+0000","last_became_peered":"2024-03-26T21:48:58.975714+0000","last_unstale":"2024-03-26T21:50:09.976614+0000","last_undegraded":"2024-03-26T21:50:09.976614+0000","last_fullsized":"2024-03-26T21:50:09.976614+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":598,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964037+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:50:09.964037+0000","last_peered":"2024-03-26T21:50:09.964037+0000","last_clean":"2024-03-26T21:50:09.964037+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:50:09.964037+0000","last_undegraded":"2024-03-26T21:50:09.964037+0000","last_fullsized":"2024-03-26T21:50:09.964037+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":253,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.975094+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:50:09.975094+0000","last_peered":"2024-03-26T21:50:09.975094+0000","last_clean":"2024-03-26T21:50:09.975094+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:50:09.975094+0000","last_undegraded":"2024-03-26T21:50:09.975094+0000","last_fullsized":"2024-03-26T21:50:09.975094+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":642,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964199+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:50:09.964199+0000","last_peered":"2024-03-26T21:50:09.964199+0000","last_clean":"2024-03-26T21:50:09.964199+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:50:09.964199+0000","last_undegraded":"2024-03-26T21:50:09.964199+0000","last_fullsized":"2024-03-26T21:50:09.964199+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":907,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-26T21:50:06.865431+0000","last_change":"2024-03-26T21:48:58.978607+0000","last_active":"2024-03-26T21:50:06.865431+0000","last_peered":"2024-03-26T21:50:06.865431+0000","last_clean":"2024-03-26T21:50:06.865431+0000","last_became_active":"2024-03-26T21:48:58.978253+0000","last_became_peered":"2024-03-26T21:48:58.978253+0000","last_unstale":"2024-03-26T21:50:06.865431+0000","last_undegraded":"2024-03-26T21:50:06.865431+0000","last_fullsized":"2024-03-26T21:50:06.865431+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:33.984501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.984501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":910,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.970996+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:50:09.970996+0000","last_peered":"2024-03-26T21:50:09.970996+0000","last_clean":"2024-03-26T21:50:09.970996+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:50:09.970996+0000","last_undegraded":"2024-03-26T21:50:09.970996+0000","last_fullsized":"2024-03-26T21:50:09.970996+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":606,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-26T21:50:09.964911+0000","last_change":"2024-03-26T21:48:44.829403+0000","last_active":"2024-03-26T21:50:09.964911+0000","last_peered":"2024-03-26T21:50:09.964911+0000","last_clean":"2024-03-26T21:50:09.964911+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:50:09.964911+0000","last_undegraded":"2024-03-26T21:50:09.964911+0000","last_fullsized":"2024-03-26T21:50:09.964911+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.829181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.829181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4853399576,"num_objects":74058,"num_object_clones":0,"num_object_copies":222174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2415,"num_objects_misplaced":7334,"num_objects_unfound":0,"num_objects_dirty":74058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163182,"num_write_kb":7591617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28881,"num_bytes_recovered":1892745216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15330717696,"data_stored":15330705480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74710,"ondisk_log_size":74710,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1177,"num_read_kb":1172,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":7130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1072,"ondisk_log_size":1072,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973803,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2679688,"kb_used_data":2120704,"kb_used_omap":6,"kb_used_meta":558841,"kb_avail":91065464,"statfs":{"total":95995035648,"available":93251035136,"internally_reserved":0,"allocated":2171600896,"data_stored":2171178662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":572253480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365070,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3117696,"kb_used_data":2414584,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":90627456,"statfs":{"total":95995035648,"available":92802514944,"internally_reserved":0,"allocated":2472534016,"data_stored":2472118054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575144,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3667608,"kb_used_data":2729872,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":90077544,"statfs":{"total":95995035648,"available":92239405056,"internally_reserved":0,"allocated":2795388928,"data_stored":2794949131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2798668,"kb_used_data":1878848,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":90946484,"statfs":{"total":95995035648,"available":93129199616,"internally_reserved":0,"allocated":1923940352,"data_stored":1923510807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149980,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3018256,"kb_used_data":2076360,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":90726896,"statfs":{"total":95995035648,"available":92904341504,"internally_reserved":0,"allocated":2126192640,"data_stored":2125760532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":964491719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650576,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322640,"kb_used_data":1536644,"kb_used_omap":4,"kb_used_meta":785979,"kb_avail":91422512,"statfs":{"total":95995035648,"available":93616652288,"internally_reserved":0,"allocated":1573523456,"data_stored":1573096146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":804843171},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659409,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1931196,"kb_used_data":1205876,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":91813956,"statfs":{"total":95995035648,"available":94017490944,"internally_reserved":0,"allocated":1234817024,"data_stored":1234404916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1911095296,"data_stored":1911095296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2782072832,"data_stored":2782068760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1140260864,"data_stored":1140260864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2099118080,"data_stored":2099118080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1553731584,"data_stored":1553727512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1227227136,"data_stored":1227227136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2456096768,"data_stored":2456092696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2161115136,"data_stored":2161115136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:50:14.892 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-26T21:50:14.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:14.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:15.044 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:15 smithi055 ceph-mon[29406]: pgmap v1030: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 2415/222768 objects degraded (1.084%); 7334/222768 objects misplaced (3.292%); 14 MiB/s, 221 objects/s recovering 2024-03-26T21:50:15.044 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:15 smithi055 ceph-mon[29406]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:15.045 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:15 smithi055 ceph-mon[40587]: pgmap v1030: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 2415/222768 objects degraded (1.084%); 7334/222768 objects misplaced (3.292%); 14 MiB/s, 221 objects/s recovering 2024-03-26T21:50:15.045 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:15 smithi055 ceph-mon[40587]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:15.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:15.100 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:15.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:15 smithi067 ceph-mon[33346]: pgmap v1030: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 2415/222768 objects degraded (1.084%); 7334/222768 objects misplaced (3.292%); 14 MiB/s, 221 objects/s recovering 2024-03-26T21:50:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:15 smithi067 ceph-mon[33346]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:15.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:15.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:15.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:50:15.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:15.476 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:15.497 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:15.502 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:15.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:15.628 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:15.681 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:15.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:15.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:15.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:15.881 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:15.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:15.983 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:16.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:16.082 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:16 smithi067 ceph-mon[33346]: osdmap e481: 8 total, 8 up, 7 in 2024-03-26T21:50:16.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.151 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:16.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:16.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:16.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:16 smithi055 ceph-mon[29406]: osdmap e481: 8 total, 8 up, 7 in 2024-03-26T21:50:16.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:16 smithi055 ceph-mon[40587]: osdmap e481: 8 total, 8 up, 7 in 2024-03-26T21:50:16.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.428 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:16.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:16.529 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.567 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:16.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:16.668 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:16.711 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:16.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:16.855 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:16.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:16.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:17.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:17.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:17.232 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:17 smithi067 ceph-mon[33346]: pgmap v1032: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2415/221859 objects degraded (1.089%); 7334/221859 objects misplaced (3.306%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:50:17.232 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:17 smithi067 ceph-mon[33346]: osdmap e482: 8 total, 8 up, 7 in 2024-03-26T21:50:17.232 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:17 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2415/221859 objects degraded (1.089%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:17.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:17.282 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[40587]: pgmap v1032: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2415/221859 objects degraded (1.089%); 7334/221859 objects misplaced (3.306%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:50:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[40587]: osdmap e482: 8 total, 8 up, 7 in 2024-03-26T21:50:17.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2415/221859 objects degraded (1.089%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[29406]: pgmap v1032: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 2415/221859 objects degraded (1.089%); 7334/221859 objects misplaced (3.306%); 13 MiB/s, 203 objects/s recovering 2024-03-26T21:50:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[29406]: osdmap e482: 8 total, 8 up, 7 in 2024-03-26T21:50:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:17 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2415/221859 objects degraded (1.089%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:17.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:17.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:17.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:50:17.517 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:17.555 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:17.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:17.710 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:17.809 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:17.809 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:17.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:17.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:17.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:17.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:18.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:18.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:18.146 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:18.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.282 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:18.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:18.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:18.521 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:18.556 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:18.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:18.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:50:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:18.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:18.809 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:18.861 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:18.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: pgmap v1034: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 737 op/s; 2569/219444 objects degraded (1.171%); 7023/219444 objects misplaced (3.200%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:50:18.901 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:50:18.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:19.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:19.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: pgmap v1034: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 737 op/s; 2569/219444 objects degraded (1.171%); 7023/219444 objects misplaced (3.200%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:18 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:50:19.228 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: pgmap v1034: 148 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 737 op/s; 2569/219444 objects degraded (1.171%); 7023/219444 objects misplaced (3.200%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:50:19.229 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:19.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:19.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:19.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:19.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:19.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:19.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:19.710 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:19.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.848 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:19.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:19.982 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:19.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:20.056 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:50:20.058 INFO:teuthology.orchestra.run.smithi055.stderr:set pool 1 pgp_num to 1 2024-03-26T21:50:20.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.127 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:20.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:20.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:20.265 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[29406]: osdmap e483: 8 total, 8 up, 7 in 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[40587]: osdmap e483: 8 total, 8 up, 7 in 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:50:20.306 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:20.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:20 smithi067 ceph-mon[33346]: osdmap e483: 8 total, 8 up, 7 in 2024-03-26T21:50:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T21:50:20.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:20.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:20.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:20.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.685 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:20.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:20.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:20.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:20.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:21.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:21.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:21.194 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[40587]: pgmap v1036: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 2659/208866 objects degraded (1.273%); 6570/208866 objects misplaced (3.146%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T21:50:21.194 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:21.194 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:50:21.194 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[40587]: osdmap e484: 8 total, 8 up, 7 in 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[29406]: pgmap v1036: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 2659/208866 objects degraded (1.273%); 6570/208866 objects misplaced (3.146%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:21.195 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:21 smithi055 ceph-mon[29406]: osdmap e484: 8 total, 8 up, 7 in 2024-03-26T21:50:21.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:21 smithi067 ceph-mon[33346]: pgmap v1036: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 772 op/s; 2659/208866 objects degraded (1.273%); 6570/208866 objects misplaced (3.146%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T21:50:21.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:21 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:21.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2645097727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T21:50:21.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3802051070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:21.222 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:21 smithi067 ceph-mon[33346]: osdmap e484: 8 total, 8 up, 7 in 2024-03-26T21:50:21.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.259 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:21.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:21.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:21.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:21.436 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:21.436 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:21.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:21.535 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:21.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:21.671 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:21.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.810 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:21.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:21.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:21.956 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:21.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:22.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:22.103 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:22.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:50:22.204 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.245 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:22.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:22.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2659/208866 objects degraded (1.273%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:22.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2659/208866 objects degraded (1.273%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:22.346 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:22.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:22.446 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:22.447 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:22.447 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:22.461 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:22.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2659/208866 objects degraded (1.273%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:22.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:22.500 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:22.540 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:22.581 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:22.624 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:22.628 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:22.629 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:22.629 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:22.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:22.678 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:22.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:22.919 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:22.954 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:22.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:23.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:23.096 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:23.198 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:23.229 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:23.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:23 smithi055 ceph-mon[29406]: pgmap v1038: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2799/206943 objects degraded (1.353%); 6295/206943 objects misplaced (3.042%); 6.8 MiB/s, 108 objects/s recovering 2024-03-26T21:50:23.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:23 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1876960772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:23.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:23 smithi055 ceph-mon[40587]: pgmap v1038: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2799/206943 objects degraded (1.353%); 6295/206943 objects misplaced (3.042%); 6.8 MiB/s, 108 objects/s recovering 2024-03-26T21:50:23.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:23 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1876960772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:23.325 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:23 smithi067 ceph-mon[33346]: pgmap v1038: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2799/206943 objects degraded (1.353%); 6295/206943 objects misplaced (3.042%); 6.8 MiB/s, 108 objects/s recovering 2024-03-26T21:50:23.326 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:23 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1876960772' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:23.331 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:23.366 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:23.468 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:23.513 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:23.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:23.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:23.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:23.930 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:24.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:24.075 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:24.177 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:24.203 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:24.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:24.338 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:24.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:24.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:24.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:24.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:24.758 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:24.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:24.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:25.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:25.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:25.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:25.175 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:25.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:25.308 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:25.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:25.422 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[40587]: pgmap v1039: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 3019/193962 objects degraded (1.556%); 5954/193962 objects misplaced (3.070%); 6.3 MiB/s, 100 objects/s recovering 2024-03-26T21:50:25.422 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:50:25.422 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:50:25.423 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[29406]: pgmap v1039: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 3019/193962 objects degraded (1.556%); 5954/193962 objects misplaced (3.070%); 6.3 MiB/s, 100 objects/s recovering 2024-03-26T21:50:25.423 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:50:25.423 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:25 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:50:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:25 smithi067 ceph-mon[33346]: pgmap v1039: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 3019/193962 objects degraded (1.556%); 5954/193962 objects misplaced (3.070%); 6.3 MiB/s, 100 objects/s recovering 2024-03-26T21:50:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:25 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:50:25.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:25 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:50:25.536 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:50:25.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:25.691 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:25.731 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:25.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:25.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:25.959 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:25.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:25.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:26.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.139 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:26.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:26.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.274 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:26.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:26.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:26.402 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:26.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:26 smithi067 ceph-mon[33346]: osdmap e485: 8 total, 8 up, 7 in 2024-03-26T21:50:26.504 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:26.528 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:26.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:26 smithi055 ceph-mon[29406]: osdmap e485: 8 total, 8 up, 7 in 2024-03-26T21:50:26.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:26 smithi055 ceph-mon[40587]: osdmap e485: 8 total, 8 up, 7 in 2024-03-26T21:50:26.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:26.682 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:50:26.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:50:26.682 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T21:50:26.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T21:50:26.770 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:26.817 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:26.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:26.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:26.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:27.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:27.094 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:27.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:27.235 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:27.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.376 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:27.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:27.462 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:27 smithi067 ceph-mon[33346]: pgmap v1041: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 832 op/s; 3019/192915 objects degraded (1.565%); 5954/192915 objects misplaced (3.086%); 2.4 MiB/s, 38 objects/s recovering 2024-03-26T21:50:27.462 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 3019/192915 objects degraded (1.565%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:27.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:27.503 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:27 smithi055 ceph-mon[29406]: pgmap v1041: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 832 op/s; 3019/192915 objects degraded (1.565%); 5954/192915 objects misplaced (3.086%); 2.4 MiB/s, 38 objects/s recovering 2024-03-26T21:50:27.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 3019/192915 objects degraded (1.565%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:27 smithi055 ceph-mon[40587]: pgmap v1041: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 0 B/s wr, 832 op/s; 3019/192915 objects degraded (1.565%); 5954/192915 objects misplaced (3.086%); 2.4 MiB/s, 38 objects/s recovering 2024-03-26T21:50:27.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 3019/192915 objects degraded (1.565%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:27.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:27.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:27.777 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:27.879 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:27.904 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:27.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:27.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:28.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:28.041 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:28.079 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:50:28.079 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:50:28.079 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:50:28.079 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:50:28.079 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:50:28.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:28.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:28.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:28.415 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:28.450 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:28.552 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:28.590 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:28.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:28.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:28.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:28.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:28.994 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:28.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:29.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:29.205 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:29.215 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:29.232 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:29 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:29 smithi067 ceph-mon[33346]: pgmap v1042: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 792 op/s; 3211/189858 objects degraded (1.691%); 5621/189858 objects misplaced (2.961%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.278 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:29.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:29.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:29 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:29.307 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:29 smithi055 ceph-mon[29406]: pgmap v1042: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 792 op/s; 3211/189858 objects degraded (1.691%); 5621/189858 objects misplaced (2.961%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:29.307 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:29.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:29 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:29.308 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:29 smithi055 ceph-mon[40587]: pgmap v1042: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 792 op/s; 3211/189858 objects degraded (1.691%); 5621/189858 objects misplaced (2.961%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:29.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:29.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:29.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.568 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:29.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:29.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:29.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:29.849 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:29.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:29.990 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:29.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:30.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.132 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:30.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:30.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:30.271 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:30.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:30.409 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:30.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:30.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:30.687 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:30.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.813 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:30.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:30.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:30.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:31.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.092 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:31.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:31.124 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:50:31.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:31.229 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:31.293 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:31 smithi055 ceph-mon[40587]: pgmap v1043: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 3443/177303 objects degraded (1.942%); 5284/177303 objects misplaced (2.980%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:31.293 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:31 smithi055 ceph-mon[29406]: pgmap v1043: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 3443/177303 objects degraded (1.942%); 5284/177303 objects misplaced (2.980%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:31.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:31.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:31.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:31 smithi067 ceph-mon[33346]: pgmap v1043: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 3443/177303 objects degraded (1.942%); 5284/177303 objects misplaced (2.980%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:31.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:31.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:31.647 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:31.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:31.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:31.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:31.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:50:31.921 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:31.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:32.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.075 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:32.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:32.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:32.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:32.327 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:32.354 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:32.360 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:32.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:32.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:32 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 3443/177303 objects degraded (1.942%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:32.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:32.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:32 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 3443/177303 objects degraded (1.942%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:32.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:32 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 3443/177303 objects degraded (1.942%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:32.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.645 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:32.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:32.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:32.784 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:32.886 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:32.933 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:32.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:33.035 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:33.069 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:33.171 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:33.201 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:33.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:33.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:33 smithi067 ceph-mon[33346]: pgmap v1044: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 3579/175494 objects degraded (2.039%); 4995/175494 objects misplaced (2.846%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:33.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:33 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:33 smithi055 ceph-mon[40587]: pgmap v1044: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 3579/175494 objects degraded (2.039%); 4995/175494 objects misplaced (2.846%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:33.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:33 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:33 smithi055 ceph-mon[29406]: pgmap v1044: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 3579/175494 objects degraded (2.039%); 4995/175494 objects misplaced (2.846%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:33.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:33 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:33.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.611 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:33.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:33.712 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:33.745 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:33.847 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:33.885 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:33.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:33.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:34.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:34.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:34.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:34.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:34.434 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:34.462 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147853 9.1 9:8b320b50:::benchmark_data_smithi055_135863_object17202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 568.75728847100004, 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.068236535, 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147853 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709526+0000", 2024-03-26T21:50:34.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709527+0000", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3900000000000003e-07 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709530+0000", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3950000000000001e-06 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709538+0000", 2024-03-26T21:50:34.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4610000000000002e-06 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709543+0000", 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2839999999999997e-06 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709581+0000", 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8556999999999997e-05 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.709587+0000", 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5509999999999999e-06 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665317+0000", 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95572950800000001 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665332+0000", 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5940999999999999e-05 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.770155+0000", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.104822763 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.771678+0000", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001522949 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.771834+0000", 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015530700000000001 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773789+0000", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019550790000000002 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775042+0000", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001252925 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777663+0000", 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002621717 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777700+0000", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6646000000000001e-05 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777762+0000", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2513000000000003e-05 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:147865 9.1 9:83cb768e:::benchmark_data_smithi055_135863_object17214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:50:34.470 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 568.73700606199998, 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0480903599999998, 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147865 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729808+0000", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729810+0000", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3209999999999999e-06 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729814+0000", 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6330000000000004e-06 2024-03-26T21:50:34.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729826+0000", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2021000000000001e-05 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729833+0000", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8650000000000003e-06 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729879+0000", 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6397000000000003e-05 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:05.729886+0000", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4540000000000004e-06 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665361+0000", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93547465600000002 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:06.665369+0000", 2024-03-26T21:50:34.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0639999999999994e-06 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773836+0000", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.108466988 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773917+0000", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1817999999999995e-05 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.773990+0000", 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2677999999999998e-05 2024-03-26T21:50:34.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775202+0000", 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001211424 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.775237+0000", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4974000000000001e-05 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777796+0000", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025594179999999999 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777832+0000", 2024-03-26T21:50:34.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6492999999999998e-05 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:07.777899+0000", 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6155999999999995e-05 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:166763 9.1 9:8eba03ed:::benchmark_data_smithi055_135863_object36112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 549.88466271000004, 2024-03-26T21:50:34.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.149474361, 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 166763 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582152+0000", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582153+0000", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4270000000000001e-06 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582159+0000", 2024-03-26T21:50:34.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.049e-06 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582175+0000", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5739999999999998e-05 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582184+0000", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.772e-06 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582242+0000", 2024-03-26T21:50:34.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8118999999999997e-05 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582410+0000", 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000167897 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4", 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.582557+0000", 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014689600000000001 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.610257+0000", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027700675000000001 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:24.610271+0000", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4199000000000001e-05 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731535+0000", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1212639820000001 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:41:25.731626+0000", 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0605000000000004e-05 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241067 9.1 9:89e6b8a5:::benchmark_data_smithi055_135863_object110416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.795862+0000", 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.670952785, 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0772418400000001, 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.484 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241067 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795862+0000", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795862+0000", 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795862+0000", 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0500000000000002e-07 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795866+0000", 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7409999999999998e-06 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795883+0000", 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7252999999999999e-05 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.795892+0000", 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1519999999999992e-06 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.796063+0000", 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171177 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.796202+0000", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138806 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,4,7", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.796352+0000", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014998100000000001 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.802487+0000", 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061357310000000002 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.802501+0000", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3638000000000001e-05 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.864912+0000", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.062410657 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for missing object", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.865006+0000", 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4065000000000001e-05 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873003+0000", 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079975389999999997 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873103+0000", 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100195 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241092 9.1 9:8012a5c3:::benchmark_data_smithi055_135863_object110441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.902257+0000", 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.56455760300003, 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97377168300000005, 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241092 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902257+0000", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902257+0000", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902258+0000", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5009999999999999e-06 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902263+0000", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0030000000000002e-06 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902276+0000", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2727e-05 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902284+0000", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5800000000000003e-06 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902342+0000", 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8559e-05 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.902351+0000", 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6989999999999992e-06 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.865321+0000", 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96297051300000003 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.865456+0000", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013489799999999999 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.865565+0000", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010895699999999999 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.872589+0000", 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070235940000000002 2024-03-26T21:50:34.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.872650+0000", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1137999999999996e-05 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.875899+0000", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003248974 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.875955+0000", 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5955999999999998e-05 2024-03-26T21:50:34.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876028+0000", 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3584000000000006e-05 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241113 9.1 9:83199cb6:::benchmark_data_smithi055_135863_object110462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.915715+0000", 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.55109898799998, 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.96057977999999999, 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.497 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241113 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915715+0000", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915715+0000", 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915717+0000", 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.44e-06 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915723+0000", 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9340000000000003e-06 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915734+0000", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1476e-05 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915742+0000", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1089999999999998e-06 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915835+0000", 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2787000000000005e-05 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.915846+0000", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0535e-05 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.867754+0000", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.951908015 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.867860+0000", 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106066 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.867967+0000", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107781 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873094+0000", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051266230000000003 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873185+0000", 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0587000000000005e-05 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876175+0000", 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029902930000000002 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876230+0000", 2024-03-26T21:50:34.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5006999999999997e-05 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876295+0000", 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5127000000000002e-05 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241125 9.1 9:8dd9c813:::benchmark_data_smithi055_135863_object110474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.924539+0000", 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.54227543799999, 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95189058699999995, 2024-03-26T21:50:34.504 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241125 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924539+0000", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924539+0000", 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924541+0000", 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.928e-06 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924547+0000", 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6800000000000004e-06 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924562+0000", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4105e-05 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924570+0000", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6410000000000008e-06 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924634+0000", 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4113000000000001e-05 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.924645+0000", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0811e-05 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.869931+0000", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94528614300000002 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.870065+0000", 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013337399999999999 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.870200+0000", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013509399999999999 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873102+0000", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002901776 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873236+0000", 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013456699999999999 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876336+0000", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00309939 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876366+0000", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0383e-05 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876429+0000", 2024-03-26T21:50:34.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3582000000000002e-05 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241132 9.1 9:888433be:::benchmark_data_smithi055_135863_object110481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.928333+0000", 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.53848087699998, 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94828721299999996, 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241132 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928333+0000", 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928333+0000", 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928335+0000", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.522e-06 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928345+0000", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5400000000000001e-06 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928360+0000", 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968e-05 2024-03-26T21:50:34.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928370+0000", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4900000000000006e-06 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928429+0000", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9012999999999999e-05 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.928438+0000", 2024-03-26T21:50:34.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1970000000000002e-06 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873272+0000", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94483356100000004 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873379+0000", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010690499999999999 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.873459+0000", 2024-03-26T21:50:34.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0111000000000006e-05 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876441+0000", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002982476 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876477+0000", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5423000000000003e-05 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876515+0000", 2024-03-26T21:50:34.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8299e-05 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876543+0000", 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7926e-05 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876621+0000", 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7781999999999995e-05 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241138 9.1 9:80f6bd82:::benchmark_data_smithi055_135863_object110487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.930576+0000", 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.53623805299998, 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95567765199999999, 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241138 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930576+0000", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930576+0000", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930577+0000", 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0589999999999999e-06 2024-03-26T21:50:34.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930583+0000", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4709999999999998e-06 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930593+0000", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0502999999999999e-05 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930601+0000", 2024-03-26T21:50:34.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7789999999999996e-06 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930664+0000", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2697e-05 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.930672+0000", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5420000000000005e-06 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876670+0000", 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94599761699999996 2024-03-26T21:50:34.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876771+0000", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101394 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.876992+0000", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022042000000000001 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.885975+0000", 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008983474 2024-03-26T21:50:34.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886004+0000", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8377999999999998e-05 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886069+0000", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5400000000000004e-05 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886130+0000", 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1341999999999996e-05 2024-03-26T21:50:34.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886254+0000", 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123576 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.25220.0:241147 9.1 9:8e4c33dc:::benchmark_data_smithi055_135863_object110496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:42:48.936985+0000", 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 465.529829345, 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.949582598, 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.524 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.25220", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 241147 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.936985+0000", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.936985+0000", 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.936987+0000", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9630000000000001e-06 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.936998+0000", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1457000000000001e-05 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.937020+0000", 2024-03-26T21:50:34.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1999000000000001e-05 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.937036+0000", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5854000000000001e-05 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.937104+0000", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7451000000000002e-05 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:48.937115+0000", 2024-03-26T21:50:34.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1256e-05 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.879213+0000", 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.94209779699999996 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.879359+0000", 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014585499999999999 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4,7", 2024-03-26T21:50:34.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.879491+0000", 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013254099999999999 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886016+0000", 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065247439999999999 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886320+0000", 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030399300000000001 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886370+0000", 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9823000000000001e-05 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886455+0000", 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5154999999999994e-05 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:42:49.886568+0000", 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011271 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.531 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52076 10.7 10:e2821005:::benchmark_data_smithi055_163313_object52075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337386+0000", 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12942827099999, 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0234862579999999, 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52076 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337386+0000", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337386+0000", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337388+0000", 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6950000000000001e-06 2024-03-26T21:50:34.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337438+0000", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0229000000000003e-05 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337455+0000", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6603999999999999e-05 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337465+0000", 2024-03-26T21:50:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0635000000000001e-05 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340509+0000", 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003043647 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340516+0000", 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6590000000000001e-06 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409390+0000", 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068874605000000005 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409399+0000", 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1540000000000008e-06 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360788+0000", 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.951388448 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.360872+0000", 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4581999999999998e-05 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52077 10.7 10:e579721c:::benchmark_data_smithi055_163313_object52076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337484+0000", 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12933035999998, 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.023859141, 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52077 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337484+0000", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337484+0000", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337484+0000", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9900000000000001e-07 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337520+0000", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.506e-05 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337527+0000", 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7130000000000005e-06 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337534+0000", 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.55e-06 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340560+0000", 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030259240000000001 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340564+0000", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7129999999999996e-06 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409425+0000", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068860418000000007 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409432+0000", 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1849999999999998e-06 2024-03-26T21:50:34.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.361216+0000", 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95178437400000004 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.361343+0000", 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012670499999999999 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52079 10.17 10:e98621f3:::benchmark_data_smithi055_163313_object52078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337543+0000", 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12927099799998, 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0321875309999999, 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52079 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337543+0000", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337543+0000", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337544+0000", 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-03-26T21:50:34.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337582+0000", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8640999999999998e-05 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337590+0000", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337597+0000", 2024-03-26T21:50:34.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0090000000000001e-06 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.412350+0000", 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.074752932999999994 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.412357+0000", 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2629999999999997e-06 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.369626+0000", 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95726823800000005 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.369731+0000", 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105289 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52082 10.7 10:e273e16f:::benchmark_data_smithi055_163313_object52081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337606+0000", 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12920828099999, 2024-03-26T21:50:34.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0238429339999999, 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52082 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337606+0000", 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337606+0000", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337607+0000", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9100000000000004e-07 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337644+0000", 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7843999999999999e-05 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337652+0000", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.233e-06 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337658+0000", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4280000000000001e-06 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340600+0000", 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002942273 2024-03-26T21:50:34.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340605+0000", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4089999999999996e-06 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409458+0000", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068853257000000001 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409466+0000", 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6709999999999999e-06 2024-03-26T21:50:34.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.361377+0000", 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95191176600000005 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.361449+0000", 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1562000000000004e-05 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.552 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52083 10.7 10:e7b80273:::benchmark_data_smithi055_163313_object52082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337667+0000", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12914729599999, 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0357305029999999, 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52083 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337667+0000", 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337667+0000", 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337668+0000", 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337702+0000", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4884e-05 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337710+0000", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.221e-06 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337716+0000", 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8540000000000002e-06 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340625+0000", 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029093999999999999 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340629+0000", 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.459e-06 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409492+0000", 2024-03-26T21:50:34.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068862781999999997 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.409499+0000", 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.345e-06 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.361864+0000", 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95236401299999995 2024-03-26T21:50:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.362608+0000", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074409499999999996 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.373320+0000", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010712597000000001 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.373398+0000", 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7328000000000002e-05 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:52092 10.7 10:e1fd3a25:::benchmark_data_smithi055_163313_object52091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:44:04.337724+0000", 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 390.12909018300002, 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0395397340000001, 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 52092 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337724+0000", 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337724+0000", 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337725+0000", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3000000000000001e-07 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337758+0000", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467000000000003e-05 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337764+0000", 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.443e-06 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.337770+0000", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9549999999999997e-06 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340650+0000", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002879932 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting_for_map not empty", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.340655+0000", 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.391e-06 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.412158+0000", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.071503351000000007 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:04.412173+0000", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4688000000000001e-05 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.362790+0000", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95061691100000001 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-03-26T21:50:34.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.363423+0000", 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00063309 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.377068+0000", 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013645465000000001 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:44:05.377264+0000", 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019561099999999999 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:111078 10.7 10:e1870935:::benchmark_data_smithi055_163313_object7029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:45:19.457201+0000", 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 315.00961329299997, 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97146038599999995, 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111078 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457201+0000", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457201+0000", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457202+0000", 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.46e-06 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457213+0000", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0814e-05 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457227+0000", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4038e-05 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457236+0000", 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457310+0000", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3188999999999993e-05 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:19.457322+0000", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2283000000000001e-05 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.414813+0000", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.95749112599999997 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.414912+0000", 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8816999999999999e-05 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3,5", 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.414984+0000", 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1860000000000007e-05 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.418883+0000", 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038989799999999998 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.418933+0000", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0890000000000002e-05 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.428583+0000", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00964981 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.428617+0000", 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3479000000000003e-05 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:45:20.428661+0000", 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4653000000000001e-05 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:143261 10.f 10:f4d3e0ed:::benchmark_data_smithi055_163313_object39212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:11.808046+0000", 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 262.65876795399998, 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.068395787, 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 143261 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808046+0000", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808046+0000", 2024-03-26T21:50:34.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808047+0000", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6600000000000005e-07 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808051+0000", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2150000000000003e-06 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808062+0000", 2024-03-26T21:50:34.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0057999999999999e-05 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808069+0000", 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8709999999999997e-06 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808136+0000", 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6534999999999999e-05 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808262+0000", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012611499999999999 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.808393+0000", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013051 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.811494+0000", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031016239999999999 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.811512+0000", 2024-03-26T21:50:34.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7538999999999999e-05 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.811904+0000", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392407 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.852660+0000", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040756368000000001 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.852673+0000", 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2156e-05 2024-03-26T21:50:34.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.876367+0000", 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.023694745 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.876442+0000", 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4777999999999995e-05 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.578 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:143334 10.f 10:fb0f4538:::benchmark_data_smithi055_163313_object39285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:11.895353+0000", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 262.57146158199998, 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.999155142, 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 143334 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895353+0000", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895353+0000", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895354+0000", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0299999999999997e-07 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895361+0000", 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8369999999999997e-06 2024-03-26T21:50:34.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895370+0000", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895376+0000", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4140000000000002e-06 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895427+0000", 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0714999999999998e-05 2024-03-26T21:50:34.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.895436+0000", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6949999999999995e-06 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.879124+0000", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.98368741000000004 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.879231+0000", 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107027 2024-03-26T21:50:34.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.879340+0000", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108935 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.888210+0000", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088703489999999996 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.888252+0000", 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2348999999999997e-05 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894376+0000", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061241029999999997 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894464+0000", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7721999999999998e-05 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894508+0000", 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3733000000000003e-05 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15609.0:143341 10.f 10:fcd7040d:::benchmark_data_smithi055_163313_object39292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:46:11.903916+0000", 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 262.56289786399998, 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.99077614199999997, 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15609", 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 143341 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:34.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903916+0000", 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903916+0000", 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:34.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903917+0000", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4900000000000005e-07 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903922+0000", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9899999999999997e-06 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903934+0000", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.129e-05 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:34.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.903941+0000", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3379999999999999e-06 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.904017+0000", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6217999999999994e-05 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:11.904034+0000", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6388e-05 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.882203+0000", 2024-03-26T21:50:34.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.97816940399999996 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.882317+0000", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114393 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2,3", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.882424+0000", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106822 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894245+0000", 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011820884 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894516+0000", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027093000000000001 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894569+0000", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3050000000000002e-05 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894651+0000", 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1866000000000005e-05 2024-03-26T21:50:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:46:12.894693+0000", 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.172e-05 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:34.591 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:50:34.629 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:34.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:34.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:34.899 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:34.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:35.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:35.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:35.171 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:35.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:35.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:35.451 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:35.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:35 smithi055 ceph-mon[40587]: pgmap v1045: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 3794/162864 objects degraded (2.330%); 4664/162864 objects misplaced (2.864%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:35 smithi067 ceph-mon[33346]: pgmap v1045: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 3794/162864 objects degraded (2.330%); 4664/162864 objects misplaced (2.864%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:35.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:35 smithi055 ceph-mon[29406]: pgmap v1045: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 3794/162864 objects degraded (2.330%); 4664/162864 objects misplaced (2.864%); 3.2 MiB/s, 51 objects/s recovering 2024-03-26T21:50:35.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:35.590 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.594 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:35.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:35.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:50:35.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:35.753 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:35.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:35.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:35.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:35.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.046 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:36.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:36.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:36.186 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:36.288 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.329 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:36.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:36.430 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:36.469 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:36.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:36.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:36.745 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:36.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:36.876 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:36.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:36.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:37.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:37.152 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:37.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:37.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:37.423 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:37 smithi067 ceph-mon[33346]: pgmap v1046: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 959 B/s rd, 0 B/s wr, 971 op/s; 3794/161850 objects degraded (2.344%); 4664/161850 objects misplaced (2.882%); 3.0 MiB/s, 48 objects/s recovering 2024-03-26T21:50:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:37 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/544684148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:50:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 3794/161850 objects degraded (2.344%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:37.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[40587]: pgmap v1046: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 959 B/s rd, 0 B/s wr, 971 op/s; 3794/161850 objects degraded (2.344%); 4664/161850 objects misplaced (2.882%); 3.0 MiB/s, 48 objects/s recovering 2024-03-26T21:50:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/544684148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:50:37.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 3794/161850 objects degraded (2.344%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:37.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[29406]: pgmap v1046: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 959 B/s rd, 0 B/s wr, 971 op/s; 3794/161850 objects degraded (2.344%); 4664/161850 objects misplaced (2.882%); 3.0 MiB/s, 48 objects/s recovering 2024-03-26T21:50:37.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/544684148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:50:37.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 3794/161850 objects degraded (2.344%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:37.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:37.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:37.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:37.953 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:37.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:38.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:38.112 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:38.213 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:38.249 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:38.353 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:38.389 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:38.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:38.525 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:38.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:38.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:38.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:38.792 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15804.0:189526 11.d 11:bfd5c1e3:::benchmark_data_smithi055_186849_object70904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:50:38.784315+0000", 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.014797507, 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014829438, 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15804", 2024-03-26T21:50:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 189526 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784315+0000", 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784315+0000", 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784316+0000", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1939999999999999e-06 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784322+0000", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6219999999999997e-06 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784337+0000", 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.473e-05 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.784345+0000", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7290000000000006e-06 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.788242+0000", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038966859999999999 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.788372+0000", 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000129772 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.795486+0000", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071142319999999998 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.795595+0000", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109308 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.795745+0000", 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014968800000000001 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15804.0:189531 11.d 11:bdaf31c0:::benchmark_data_smithi055_186849_object70909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:50:38.789487+0000", 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0096260289999999995, 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099821040000000003, 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15804", 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 189531 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789487+0000", 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789487+0000", 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789488+0000", 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3459999999999999e-06 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789496+0000", 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8800000000000008e-06 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789511+0000", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4786e-05 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.789520+0000", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0780000000000002e-06 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.798383+0000", 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088629069999999997 2024-03-26T21:50:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.798570+0000", 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018753199999999999 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,6", 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.798771+0000", 2024-03-26T21:50:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020122900000000001 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.15804.0:189534 11.e 11:764505d6:::benchmark_data_smithi055_186849_object70912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-03-26T21:50:38.790438+0000", 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0086748899999999993, 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091069099999999993, 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.15804", 2024-03-26T21:50:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.55:0/4012274773", 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 189534 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790438+0000", 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790438+0000", 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790439+0000", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6700000000000002e-07 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790447+0000", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3899999999999993e-06 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790458+0000", 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0992000000000001e-05 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.790465+0000", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5830000000000001e-06 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.791080+0000", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00061430499999999999 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.791206+0000", 2024-03-26T21:50:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012590600000000001 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1,7", 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.791682+0000", 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000476205 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.797407+0000", 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057254469999999998 2024-03-26T21:50:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-03-26T21:50:38.797690+0000", 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000282398 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 3 2024-03-26T21:50:38.808 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:50:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:38.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:38.945 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:38.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:39.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:39.167 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:39.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:39.218 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:39.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:39.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:39.396 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:39.396 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:39 smithi067 ceph-mon[33346]: pgmap v1047: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 945 op/s; 4002/158892 objects degraded (2.519%); 4319/158892 objects misplaced (2.718%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:39.425 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:39.425 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:39 smithi055 ceph-mon[40587]: pgmap v1047: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 945 op/s; 4002/158892 objects degraded (2.519%); 4319/158892 objects misplaced (2.718%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:39.425 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:39.425 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:39 smithi055 ceph-mon[29406]: pgmap v1047: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 945 op/s; 4002/158892 objects degraded (2.519%); 4319/158892 objects misplaced (2.718%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:39.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:39.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:39.635 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:39.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:50:39.738 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:39.776 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "epoch": 485, 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "created": "2024-03-26T21:22:56.673257+0000", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "modified": "2024-03-26T21:50:25.094886+0000", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "last_up_change": "2024-03-26T21:41:24.572964+0000", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "last_in_change": "2024-03-26T21:48:57.950733+0000", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_num": 5806080, 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_set": [ 2024-03-26T21:50:39.819 INFO:teuthology.orchestra.run.smithi055.stdout: "nodeep-scrub", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "noscrub", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "pglog_hardlimit", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "purged_snapdirs", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_deletes", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "sortbitwise" 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_version": 20, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_snapshot": "", 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_max": 11, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "max_osd": 8, 2024-03-26T21:50:39.820 INFO:teuthology.orchestra.run.smithi055.stdout: "require_min_compat_client": "luminous", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "min_compat_client": "luminous", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "require_osd_release": "squid", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "allow_crimson": false, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "pools": [ 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 1, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".mgr", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:26:13.884991+0000", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.821 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 1, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 1, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 1, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 1, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 1, 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.822 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "484", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.823 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.824 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.825 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_max": 32, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_min": 1 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": {} 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 6.9800000190734863, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 6.9800000190734863, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.43000000715255737, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 3, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 3, 2024-03-26T21:50:39.826 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 2, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "datapool", 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.142617+0000", 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 8193, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.827 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 3, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 3, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 3, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 3, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 3, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.828 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "71", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 2, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 71, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.829 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.830 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.831 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd": {} 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.3299999237060547, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.3299999237060547, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.86000001430511475, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.832 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 3, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".rgw.root", 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.591599+0000", 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.833 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.834 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "68", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.835 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.836 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.837 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.5299999713897705, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.5299999713897705, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.838 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 4, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.log", 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:34.904126+0000", 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.839 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "70", 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.840 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.841 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.842 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.190000057220459, 2024-03-26T21:50:39.843 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.190000057220459, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 5, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.control", 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:36.852373+0000", 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.844 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.845 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "72", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.846 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.848 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.5299999713897705, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.5299999713897705, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.849 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 6, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.meta", 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:38.937273+0000", 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "74", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.852 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.853 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_bias": 4 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:50:39.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.75, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.75, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 11, 2024-03-26T21:50:39.855 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "unique_pool_4", 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:47:37.048063+0000", 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 16, 2024-03-26T21:50:39.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 16, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 16, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 16, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 16, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "439", 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:50:39.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:50:39.858 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:50:39.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "rados": {} 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.630000114440918, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.630000114440918, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:50:39.860 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.630000114440918, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.630000114440918, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae", 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.861 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 8, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 471, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6801", 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.862 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6802", 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6804", 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:50:39.863 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6803", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6801/317381212", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6802/317381212", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6804/317381212", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6803/317381212", 2024-03-26T21:50:39.864 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db", 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 13, 2024-03-26T21:50:39.865 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 481, 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6805", 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.866 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6806", 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6808", 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.867 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6807", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6805/3733554168", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6806/3733554168", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6808/3733554168", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6807/3733554168", 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.868 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b", 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 0, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 0, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 18, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 433, 2024-03-26T21:50:39.869 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6809", 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.870 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6810", 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.871 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6812", 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6811", 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6809/3059242923", 2024-03-26T21:50:39.872 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6810/3059242923", 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6812/3059242923", 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6811/3059242923", 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b", 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.873 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 26, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 467, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6813", 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.874 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6814", 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.875 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6816", 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6815", 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.876 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6813/3851321468", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6814/3851321468", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6816/3851321468", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6815/3851321468", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598", 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.877 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 31, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 92, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 133, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 467, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 93, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6800", 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.878 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6801", 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6803", 2024-03-26T21:50:39.879 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6802", 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6800/254752278", 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6801/254752278", 2024-03-26T21:50:39.880 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6803/254752278", 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6802/254752278", 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213", 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.881 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 41, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 467, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6804", 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.882 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6805", 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6807", 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.883 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6806", 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6804/278711358", 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6805/278711358", 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6807/278711358", 2024-03-26T21:50:39.884 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6806/278711358", 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9", 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 0, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 50, 2024-03-26T21:50:39.885 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 433, 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6808", 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.886 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6809", 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6811", 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6810", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6808/3926251639", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6809/3926251639", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6811/3926251639", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6810/3926251639", 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.888 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55", 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 59, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 248, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 276, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 476, 2024-03-26T21:50:39.889 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 249, 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6812", 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.890 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6813", 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6815", 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:50:39.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6814", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6812/650373445", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6813/650373445", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6815/650373445", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6814/650373445", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.892 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_xinfo": [ 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:50:18.453269+0000", 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.893 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:49:59.047522+0000", 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 65536, 2024-03-26T21:50:39.894 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:49:49.940173+0000", 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:49:40.789494+0000", 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:50:39.895 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:50:24.178472+0000", 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 92 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.896 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:50:06.359146+0000", 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:49:49.185772+0000", 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:50:39.897 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:49:46.380577+0000", 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 248 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap": [], 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_items": [ 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.4", 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:50:39.898 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.6", 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.18", 2024-03-26T21:50:39.899 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 0 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_primaries": [], 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_temp": [ 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "11.2", 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: 3, 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: 0 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.900 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "11.d", 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: 4, 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: 6 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_temp": [], 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklist": { 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2649740776": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/4141219490": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/147833426": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.901 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2854772543": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/2804789175": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2199694469": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1614234257": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2834138692": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1740219825": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/268664696": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/536717527": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/966571165": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2757111431": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/559658018": "2024-03-27T21:23:57.787077+0000" 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "range_blocklist": {}, 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profiles": { 2024-03-26T21:50:39.902 INFO:teuthology.orchestra.run.smithi055.stdout: "default": { 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "crush-failure-domain": "osd", 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "k": "2", 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "m": "1", 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin": "jerasure", 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "technique": "reed_sol_van" 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps_queue": [], 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "new_removed_snaps": [], 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "new_purged_snaps": [], 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_node_flags": {}, 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "device_class_flags": {}, 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode": { 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_enabled": false, 2024-03-26T21:50:39.903 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_bucket_count": 0, 2024-03-26T21:50:39.904 INFO:teuthology.orchestra.run.smithi055.stdout: "degraded_stretch_mode": 0, 2024-03-26T21:50:39.904 INFO:teuthology.orchestra.run.smithi055.stdout: "recovering_stretch_mode": 0, 2024-03-26T21:50:39.904 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_bucket": 0 2024-03-26T21:50:39.904 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:50:39.904 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:50:39.904 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:39.945 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:39.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:40.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:40.109 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:40.110 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:40.212 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.246 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:40.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:40.259 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:50:40.316 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:40 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:50:40.347 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:40.386 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:40.488 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.524 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:40.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:40.540 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:40 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:50:40.541 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:40 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:50:40.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.663 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:40.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:40.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:40.801 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:40.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:40.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:41.041 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:41.054 INFO:tasks.thrashosds.thrasher:j is {'epoch': 485, 'fsid': 'e5f52b02-ebb6-11ee-95d0-87774f69a715', 'created': '2024-03-26T21:22:56.673257+0000', 'modified': '2024-03-26T21:50:25.094886+0000', 'last_up_change': '2024-03-26T21:41:24.572964+0000', 'last_in_change': '2024-03-26T21:48:57.950733+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '484', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T21:28:33.142617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T21:28:33.591599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T21:28:34.904126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T21:28:36.852373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T21:28:38.937273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-26T21:47:37.048063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '439', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '4f487071-c151-406b-b25b-3135cb3c63ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 471, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6801', 'nonce': 317381212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6802', 'nonce': 317381212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6804', 'nonce': 317381212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6803', 'nonce': 317381212}]}, 'public_addr': '172.21.15.55:6801/317381212', 'cluster_addr': '172.21.15.55:6802/317381212', 'heartbeat_back_addr': '172.21.15.55:6804/317381212', 'heartbeat_front_addr': '172.21.15.55:6803/317381212', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6805', 'nonce': 3733554168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6806', 'nonce': 3733554168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6808', 'nonce': 3733554168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6807', 'nonce': 3733554168}]}, 'public_addr': '172.21.15.55:6805/3733554168', 'cluster_addr': '172.21.15.55:6806/3733554168', 'heartbeat_back_addr': '172.21.15.55:6808/3733554168', 'heartbeat_front_addr': '172.21.15.55:6807/3733554168', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 433, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6809', 'nonce': 3059242923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6810', 'nonce': 3059242923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6812', 'nonce': 3059242923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6811', 'nonce': 3059242923}]}, 'public_addr': '172.21.15.55:6809/3059242923', 'cluster_addr': '172.21.15.55:6810/3059242923', 'heartbeat_back_addr': '172.21.15.55:6812/3059242923', 'heartbeat_front_addr': '172.21.15.55:6811/3059242923', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '30637a8d-5c1b-4ede-8214-a4595c875d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 467, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6813', 'nonce': 3851321468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6814', 'nonce': 3851321468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6816', 'nonce': 3851321468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6815', 'nonce': 3851321468}]}, 'public_addr': '172.21.15.55:6813/3851321468', 'cluster_addr': '172.21.15.55:6814/3851321468', 'heartbeat_back_addr': '172.21.15.55:6816/3851321468', 'heartbeat_front_addr': '172.21.15.55:6815/3851321468', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'cd2d68be-4b10-4242-826b-e085c2ce5598', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 92, 'up_from': 133, 'up_thru': 467, 'down_at': 93, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 254752278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6801', 'nonce': 254752278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 254752278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 254752278}]}, 'public_addr': '172.21.15.67:6800/254752278', 'cluster_addr': '172.21.15.67:6801/254752278', 'heartbeat_back_addr': '172.21.15.67:6803/254752278', 'heartbeat_front_addr': '172.21.15.67:6802/254752278', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 467, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 278711358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6805', 'nonce': 278711358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 278711358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 278711358}]}, 'public_addr': '172.21.15.67:6804/278711358', 'cluster_addr': '172.21.15.67:6805/278711358', 'heartbeat_back_addr': '172.21.15.67:6807/278711358', 'heartbeat_front_addr': '172.21.15.67:6806/278711358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd47c79f3-96bf-46b6-a7f3-a315446d04b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 433, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 3926251639}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 3926251639}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 3926251639}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 3926251639}]}, 'public_addr': '172.21.15.67:6808/3926251639', 'cluster_addr': '172.21.15.67:6809/3926251639', 'heartbeat_back_addr': '172.21.15.67:6811/3926251639', 'heartbeat_front_addr': '172.21.15.67:6810/3926251639', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f10b98b8-887d-4f83-8bc9-99f5c302dc55', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 59, 'last_clean_end': 248, 'up_from': 276, 'up_thru': 476, 'down_at': 249, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 650373445}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6813', 'nonce': 650373445}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 650373445}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 650373445}]}, 'public_addr': '172.21.15.67:6812/650373445', 'cluster_addr': '172.21.15.67:6813/650373445', 'heartbeat_back_addr': '172.21.15.67:6815/650373445', 'heartbeat_front_addr': '172.21.15.67:6814/650373445', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:50:18.453269+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:49:59.047522+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T21:49:49.940173+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:49:40.789494+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:50:24.178472+0000', 'dead_epoch': 92}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:50:06.359146+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:49:49.185772+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:49:46.380577+0000', 'dead_epoch': 248}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.2', 'osds': [3, 0]}, {'pgid': '11.d', 'osds': [4, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.55:0/2649740776': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/4141219490': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/147833426': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2854772543': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/2804789175': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/2199694469': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/1614234257': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/2834138692': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/1740219825': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/268664696': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:6800/536717527': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/966571165': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2757111431': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/559658018': '2024-03-27T21:23:57.787077+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T21:50:41.055 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:50:41.082 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:41.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.220 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:41.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:41.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[40587]: pgmap v1048: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 4212/145596 objects degraded (2.893%); 3977/145596 objects misplaced (2.732%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[40587]: osdmap e486: 8 total, 8 up, 7 in 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[29406]: pgmap v1048: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 4212/145596 objects degraded (2.893%); 3977/145596 objects misplaced (2.732%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:50:41.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:41 smithi055 ceph-mon[29406]: osdmap e486: 8 total, 8 up, 7 in 2024-03-26T21:50:41.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:41.454 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.491 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:41.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:41.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:41.630 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:41.637 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:41 smithi067 ceph-mon[33346]: pgmap v1048: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 4212/145596 objects degraded (2.893%); 3977/145596 objects misplaced (2.732%); 3.3 MiB/s, 53 objects/s recovering 2024-03-26T21:50:41.637 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:41 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:41.637 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3368915854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:50:41.637 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:41 smithi067 ceph-mon[33346]: osdmap e486: 8 total, 8 up, 7 in 2024-03-26T21:50:41.732 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:41.762 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:41.864 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:41.919 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:41.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:42.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.072 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:42.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.211 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:42.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:42.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.348 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:42.392 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:42 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 4212/145596 objects degraded (2.893%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:42.392 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:42 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 4212/145596 objects degraded (2.893%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:42.400 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:42.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:42.472 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.479 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:42.507 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:42.533 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:42 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 4212/145596 objects degraded (2.893%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:42.543 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:42.577 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:42.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:42.613 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:42.621 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:42.650 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:42.690 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:42.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:42.729 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:42.731 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:42.731 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:42.731 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:42.731 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:42.731 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:42.732 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:42.732 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:42.732 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:42.751 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:42.752 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:42.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:42.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:43.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:43.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:43.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:43.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.471 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[29406]: pgmap v1050: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 4401/143541 objects degraded (3.066%); 3629/143541 objects misplaced (2.528%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4020935876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[40587]: pgmap v1050: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 4401/143541 objects degraded (3.066%); 3629/143541 objects misplaced (2.528%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4020935876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:50:43.529 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:43 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:50:43.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:43.611 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:43.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:43.746 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:43 smithi067 ceph-mon[33346]: pgmap v1050: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 4401/143541 objects degraded (3.066%); 3629/143541 objects misplaced (2.528%); 3.4 MiB/s, 54 objects/s recovering 2024-03-26T21:50:43.747 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4020935876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:43.747 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:43 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:50:43.747 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:43 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:50:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:43.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:50:43.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:43.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:50:43.891 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:43.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:43.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:43.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:44.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:44.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:44.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:44.351 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:44.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:44.495 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:44.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.635 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:44.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:44.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.777 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:44.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:44.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:44.906 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:44.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:45.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:45.060 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:45.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:45.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:45.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:45.314 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:50:45.321 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[29406]: pgmap v1051: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/130158 objects degraded (3.576%); 3251/130158 objects misplaced (2.498%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[40587]: pgmap v1051: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/130158 objects degraded (3.576%); 3251/130158 objects misplaced (2.498%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:50:45.342 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:45 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:50:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.346 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:45.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:45.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:45.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:45.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:45 smithi067 ceph-mon[33346]: pgmap v1051: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/130158 objects degraded (3.576%); 3251/130158 objects misplaced (2.498%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:45.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:45 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:50:45.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:45 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:50:45.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:45.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:45.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:45.984 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:46.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:46.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:46.153 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:46.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:46.294 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:46.316 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:50:46.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:46.439 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:46.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:46.553 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:46 smithi067 ceph-mon[33346]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:46.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:46 smithi055 ceph-mon[29406]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:46.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:46 smithi055 ceph-mon[40587]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:46.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:46.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:46.782 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:46.782 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:46.827 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:46.862 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:46.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:46.995 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:46.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:47.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:47.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:47.274 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:47.376 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:47.410 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:47.468 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:47 smithi067 ceph-mon[33346]: pgmap v1052: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/129216 objects degraded (3.602%); 3251/129216 objects misplaced (2.516%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:47.468 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:47 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:50:47.468 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:47 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:50:47.468 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:47 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 4654/129216 objects degraded (3.602%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:47.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:47.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:50:47.546 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:47.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[40587]: pgmap v1052: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/129216 objects degraded (3.602%); 3251/129216 objects misplaced (2.516%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:47.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:50:47.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:50:47.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 4654/129216 objects degraded (3.602%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[29406]: pgmap v1052: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4654/129216 objects degraded (3.602%); 3251/129216 objects misplaced (2.516%); 3.5 MiB/s, 55 objects/s recovering 2024-03-26T21:50:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:50:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:50:47.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:47 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 4654/129216 objects degraded (3.602%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:47.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:47.661 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1051,"stamp":"2024-03-26T21:50:43.777440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2830899648,"num_objects":43386,"num_object_clones":0,"num_object_copies":130158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4654,"num_objects_misplaced":3251,"num_objects_unfound":0,"num_objects_dirty":43386,"num_whiteouts":0,"num_read":2950,"num_read_kb":2798,"num_write":195239,"num_write_kb":7594078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31095,"num_bytes_recovered":2015175898,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75897,"ondisk_log_size":75897,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13982392,"kb_used_data":8323512,"kb_used_omap":18,"kb_used_meta":5658605,"kb_avail":642233672,"statfs":{"total":671965249536,"available":657647280128,"internally_reserved":0,"allocated":8523276288,"data_stored":8520457571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19289,"internal_metadata":5794411687},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,9,6,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-714473472,"num_objects":-10902,"num_object_clones":0,"num_object_copies":-32706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":-1413,"num_objects_unfound":0,"num_objects_dirty":-10902,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10902,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":36241408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003652"},"pg_stats":[{"pgid":"11.3","version":"486'12148","reported_seq":12274,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.548929+0000","last_change":"2024-03-26T21:49:29.997467+0000","last_active":"2024-03-26T21:50:43.548929+0000","last_peered":"2024-03-26T21:50:43.548929+0000","last_clean":"2024-03-26T21:50:43.548929+0000","last_became_active":"2024-03-26T21:49:29.980238+0000","last_became_peered":"2024-03-26T21:49:29.980238+0000","last_unstale":"2024-03-26T21:50:43.548929+0000","last_undegraded":"2024-03-26T21:50:43.548929+0000","last_fullsized":"2024-03-26T21:50:43.548929+0000","mapping_epoch":467,"log_start":"465'7477","ondisk_log_start":"465'7477","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":99,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173277184,"num_objects":2644,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'35","reported_seq":707,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262132+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:50:40.262132+0000","last_peered":"2024-03-26T21:50:40.262132+0000","last_clean":"2024-03-26T21:50:40.262132+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:50:40.262132+0000","last_undegraded":"2024-03-26T21:50:40.262132+0000","last_fullsized":"2024-03-26T21:50:40.262132+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262888+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:50:40.262888+0000","last_peered":"2024-03-26T21:50:40.262888+0000","last_clean":"2024-03-26T21:50:40.262888+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:50:40.262888+0000","last_undegraded":"2024-03-26T21:50:40.262888+0000","last_fullsized":"2024-03-26T21:50:40.262888+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"462'12","reported_seq":727,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266340+0000","last_change":"2024-03-26T21:48:59.006658+0000","last_active":"2024-03-26T21:50:40.266340+0000","last_peered":"2024-03-26T21:50:40.266340+0000","last_clean":"2024-03-26T21:50:40.266340+0000","last_became_active":"2024-03-26T21:48:58.995625+0000","last_became_peered":"2024-03-26T21:48:58.995625+0000","last_unstale":"2024-03-26T21:50:40.266340+0000","last_undegraded":"2024-03-26T21:50:40.266340+0000","last_fullsized":"2024-03-26T21:50:40.266340+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":617,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262376+0000","last_change":"2024-03-26T21:48:58.976822+0000","last_active":"2024-03-26T21:50:40.262376+0000","last_peered":"2024-03-26T21:50:40.262376+0000","last_clean":"2024-03-26T21:50:40.262376+0000","last_became_active":"2024-03-26T21:48:58.976172+0000","last_became_peered":"2024-03-26T21:48:58.976172+0000","last_unstale":"2024-03-26T21:50:40.262376+0000","last_undegraded":"2024-03-26T21:50:40.262376+0000","last_fullsized":"2024-03-26T21:50:40.262376+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"486'12292","reported_seq":16836,"reported_epoch":486,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:50:43.601655+0000","last_change":"2024-03-26T21:49:33.951482+0000","last_active":"2024-03-26T21:50:43.601655+0000","last_peered":"2024-03-26T21:50:43.601655+0000","last_clean":"2024-03-26T21:48:57.900582+0000","last_became_active":"2024-03-26T21:49:00.154091+0000","last_became_peered":"2024-03-26T21:49:00.154091+0000","last_unstale":"2024-03-26T21:50:43.601655+0000","last_undegraded":"2024-03-26T21:49:33.947342+0000","last_fullsized":"2024-03-26T21:48:59.963248+0000","mapping_epoch":462,"log_start":"466'7699","ondisk_log_start":"466'7699","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":849,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173408256,"num_objects":2646,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2615,"num_objects_misplaced":1521,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12292,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2208,"num_bytes_recovered":144703488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":1521},{"shards":"0,3","objects":-1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":703,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272256+0000","last_change":"2024-03-26T21:48:59.000246+0000","last_active":"2024-03-26T21:50:40.272256+0000","last_peered":"2024-03-26T21:50:40.272256+0000","last_clean":"2024-03-26T21:50:40.272256+0000","last_became_active":"2024-03-26T21:48:58.980284+0000","last_became_peered":"2024-03-26T21:48:58.980284+0000","last_unstale":"2024-03-26T21:50:40.272256+0000","last_undegraded":"2024-03-26T21:50:40.272256+0000","last_fullsized":"2024-03-26T21:50:40.272256+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":374,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261427+0000","last_change":"2024-03-26T21:49:29.974892+0000","last_active":"2024-03-26T21:50:40.261427+0000","last_peered":"2024-03-26T21:50:40.261427+0000","last_clean":"2024-03-26T21:50:40.261427+0000","last_became_active":"2024-03-26T21:49:29.974562+0000","last_became_peered":"2024-03-26T21:49:29.974562+0000","last_unstale":"2024-03-26T21:50:40.261427+0000","last_undegraded":"2024-03-26T21:50:40.261427+0000","last_fullsized":"2024-03-26T21:50:40.261427+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":944,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262923+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:50:40.262923+0000","last_peered":"2024-03-26T21:50:40.262923+0000","last_clean":"2024-03-26T21:50:40.262923+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:50:40.262923+0000","last_undegraded":"2024-03-26T21:50:40.262923+0000","last_fullsized":"2024-03-26T21:50:40.262923+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":740,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260813+0000","last_change":"2024-03-26T21:48:58.974425+0000","last_active":"2024-03-26T21:50:40.260813+0000","last_peered":"2024-03-26T21:50:40.260813+0000","last_clean":"2024-03-26T21:50:40.260813+0000","last_became_active":"2024-03-26T21:48:58.974080+0000","last_became_peered":"2024-03-26T21:48:58.974080+0000","last_unstale":"2024-03-26T21:50:40.260813+0000","last_undegraded":"2024-03-26T21:50:40.260813+0000","last_fullsized":"2024-03-26T21:50:40.260813+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"486'11990","reported_seq":12112,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.602046+0000","last_change":"2024-03-26T21:49:29.994417+0000","last_active":"2024-03-26T21:50:43.602046+0000","last_peered":"2024-03-26T21:50:43.602046+0000","last_clean":"2024-03-26T21:50:43.602046+0000","last_became_active":"2024-03-26T21:49:29.979423+0000","last_became_peered":"2024-03-26T21:49:29.979423+0000","last_unstale":"2024-03-26T21:50:43.602046+0000","last_undegraded":"2024-03-26T21:50:43.602046+0000","last_fullsized":"2024-03-26T21:50:43.602046+0000","mapping_epoch":467,"log_start":"465'7366","ondisk_log_start":"465'7366","created":433,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175767552,"num_objects":2682,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":469504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":652,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262411+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:50:40.262411+0000","last_peered":"2024-03-26T21:50:40.262411+0000","last_clean":"2024-03-26T21:50:40.262411+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:50:40.262411+0000","last_undegraded":"2024-03-26T21:50:40.262411+0000","last_fullsized":"2024-03-26T21:50:40.262411+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":626,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262606+0000","last_change":"2024-03-26T21:48:58.979871+0000","last_active":"2024-03-26T21:50:40.262606+0000","last_peered":"2024-03-26T21:50:40.262606+0000","last_clean":"2024-03-26T21:50:40.262606+0000","last_became_active":"2024-03-26T21:48:58.979258+0000","last_became_peered":"2024-03-26T21:48:58.979258+0000","last_unstale":"2024-03-26T21:50:40.262606+0000","last_undegraded":"2024-03-26T21:50:40.262606+0000","last_fullsized":"2024-03-26T21:50:40.262606+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.750677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":933,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273666+0000","last_change":"2024-03-26T21:49:43.657948+0000","last_active":"2024-03-26T21:50:40.273666+0000","last_peered":"2024-03-26T21:50:40.273666+0000","last_clean":"2024-03-26T21:50:40.273666+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:50:40.273666+0000","last_undegraded":"2024-03-26T21:50:40.273666+0000","last_fullsized":"2024-03-26T21:50:40.273666+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:43.657872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:43.657872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:45.879119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260472+0000","last_change":"2024-03-26T21:48:30.759102+0000","last_active":"2024-03-26T21:50:40.260472+0000","last_peered":"2024-03-26T21:50:40.260472+0000","last_clean":"2024-03-26T21:50:40.260472+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:50:40.260472+0000","last_undegraded":"2024-03-26T21:50:40.260472+0000","last_fullsized":"2024-03-26T21:50:40.260472+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.758986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.758986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"486'12089","reported_seq":12205,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.397859+0000","last_change":"2024-03-26T21:47:39.053277+0000","last_active":"2024-03-26T21:50:43.397859+0000","last_peered":"2024-03-26T21:50:43.397859+0000","last_clean":"2024-03-26T21:50:43.397859+0000","last_became_active":"2024-03-26T21:47:39.052942+0000","last_became_peered":"2024-03-26T21:47:39.052942+0000","last_unstale":"2024-03-26T21:50:43.397859+0000","last_undegraded":"2024-03-26T21:50:43.397859+0000","last_fullsized":"2024-03-26T21:50:43.397859+0000","mapping_epoch":433,"log_start":"467'7752","ondisk_log_start":"467'7752","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4337,"log_dups_size":0,"ondisk_log_size":4337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169672704,"num_objects":2589,"num_object_clones":0,"num_object_copies":7767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12089,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1080,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261080+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:50:40.261080+0000","last_peered":"2024-03-26T21:50:40.261080+0000","last_clean":"2024-03-26T21:50:40.261080+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:50:40.261080+0000","last_undegraded":"2024-03-26T21:50:40.261080+0000","last_fullsized":"2024-03-26T21:50:40.261080+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262177+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:50:40.262177+0000","last_peered":"2024-03-26T21:50:40.262177+0000","last_clean":"2024-03-26T21:50:40.262177+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:50:40.262177+0000","last_undegraded":"2024-03-26T21:50:40.262177+0000","last_fullsized":"2024-03-26T21:50:40.262177+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":790,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262472+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:50:40.262472+0000","last_peered":"2024-03-26T21:50:40.262472+0000","last_clean":"2024-03-26T21:50:40.262472+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:50:40.262472+0000","last_undegraded":"2024-03-26T21:50:40.262472+0000","last_fullsized":"2024-03-26T21:50:40.262472+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":916,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272602+0000","last_change":"2024-03-26T21:50:01.887630+0000","last_active":"2024-03-26T21:50:40.272602+0000","last_peered":"2024-03-26T21:50:40.272602+0000","last_clean":"2024-03-26T21:50:40.272602+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:50:40.272602+0000","last_undegraded":"2024-03-26T21:50:40.272602+0000","last_fullsized":"2024-03-26T21:50:40.272602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:01.887146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:50:01.887146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:20.427066+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"486'12125","reported_seq":25900,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.556460+0000","last_change":"2024-03-26T21:49:45.255955+0000","last_active":"2024-03-26T21:50:43.556460+0000","last_peered":"2024-03-26T21:50:43.556460+0000","last_clean":"2024-03-26T21:50:43.556460+0000","last_became_active":"2024-03-26T21:49:45.248068+0000","last_became_peered":"2024-03-26T21:49:45.248068+0000","last_unstale":"2024-03-26T21:50:43.556460+0000","last_undegraded":"2024-03-26T21:50:43.556460+0000","last_fullsized":"2024-03-26T21:50:43.556460+0000","mapping_epoch":471,"log_start":"465'7491","ondisk_log_start":"465'7491","created":433,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173342720,"num_objects":2645,"num_object_clones":0,"num_object_copies":7935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12125,"num_write_kb":472640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6823,"num_bytes_recovered":447152128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":630,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262759+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:50:40.262759+0000","last_peered":"2024-03-26T21:50:40.262759+0000","last_clean":"2024-03-26T21:50:40.262759+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:50:40.262759+0000","last_undegraded":"2024-03-26T21:50:40.262759+0000","last_fullsized":"2024-03-26T21:50:40.262759+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":301,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261800+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:50:40.261800+0000","last_peered":"2024-03-26T21:50:40.261800+0000","last_clean":"2024-03-26T21:50:40.261800+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:50:40.261800+0000","last_undegraded":"2024-03-26T21:50:40.261800+0000","last_fullsized":"2024-03-26T21:50:40.261800+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261247+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:50:40.261247+0000","last_peered":"2024-03-26T21:50:40.261247+0000","last_clean":"2024-03-26T21:50:40.261247+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:50:40.261247+0000","last_undegraded":"2024-03-26T21:50:40.261247+0000","last_fullsized":"2024-03-26T21:50:40.261247+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":637,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261801+0000","last_change":"2024-03-26T21:48:58.977779+0000","last_active":"2024-03-26T21:50:40.261801+0000","last_peered":"2024-03-26T21:50:40.261801+0000","last_clean":"2024-03-26T21:50:40.261801+0000","last_became_active":"2024-03-26T21:48:58.977356+0000","last_became_peered":"2024-03-26T21:48:58.977356+0000","last_unstale":"2024-03-26T21:50:40.261801+0000","last_undegraded":"2024-03-26T21:50:40.261801+0000","last_fullsized":"2024-03-26T21:50:40.261801+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:28.651656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:48:28.651656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"486'11960","reported_seq":12077,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:41.401310+0000","last_change":"2024-03-26T21:47:39.053658+0000","last_active":"2024-03-26T21:50:41.401310+0000","last_peered":"2024-03-26T21:50:41.401310+0000","last_clean":"2024-03-26T21:50:41.401310+0000","last_became_active":"2024-03-26T21:47:39.053244+0000","last_became_peered":"2024-03-26T21:47:39.053244+0000","last_unstale":"2024-03-26T21:50:41.401310+0000","last_undegraded":"2024-03-26T21:50:41.401310+0000","last_fullsized":"2024-03-26T21:50:41.401310+0000","mapping_epoch":433,"log_start":"465'7198","ondisk_log_start":"465'7198","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":471424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":340,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262271+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:50:40.262271+0000","last_peered":"2024-03-26T21:50:40.262271+0000","last_clean":"2024-03-26T21:50:40.262271+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:50:40.262271+0000","last_undegraded":"2024-03-26T21:50:40.262271+0000","last_fullsized":"2024-03-26T21:50:40.262271+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":669,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262654+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:50:40.262654+0000","last_peered":"2024-03-26T21:50:40.262654+0000","last_clean":"2024-03-26T21:50:40.262654+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:50:40.262654+0000","last_undegraded":"2024-03-26T21:50:40.262654+0000","last_fullsized":"2024-03-26T21:50:40.262654+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":927,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.117342+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:50:25.117342+0000","last_peered":"2024-03-26T21:50:25.117342+0000","last_clean":"2024-03-26T21:50:25.117342+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:50:25.117342+0000","last_undegraded":"2024-03-26T21:50:25.117342+0000","last_fullsized":"2024-03-26T21:50:25.117342+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":840,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266446+0000","last_change":"2024-03-26T21:48:58.986678+0000","last_active":"2024-03-26T21:50:40.266446+0000","last_peered":"2024-03-26T21:50:40.266446+0000","last_clean":"2024-03-26T21:50:40.266446+0000","last_became_active":"2024-03-26T21:48:58.985775+0000","last_became_peered":"2024-03-26T21:48:58.985775+0000","last_unstale":"2024-03-26T21:50:40.266446+0000","last_undegraded":"2024-03-26T21:50:40.266446+0000","last_fullsized":"2024-03-26T21:50:40.266446+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"486'11914","reported_seq":17058,"reported_epoch":486,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T21:50:41.452318+0000","last_change":"2024-03-26T21:49:36.580498+0000","last_active":"2024-03-26T21:50:41.452318+0000","last_peered":"2024-03-26T21:50:41.452318+0000","last_clean":"2024-03-26T21:48:57.955814+0000","last_became_active":"2024-03-26T21:49:00.339243+0000","last_became_peered":"2024-03-26T21:49:00.339243+0000","last_unstale":"2024-03-26T21:50:41.452318+0000","last_undegraded":"2024-03-26T21:49:36.577298+0000","last_fullsized":"2024-03-26T21:48:59.964175+0000","mapping_epoch":462,"log_start":"465'7174","ondisk_log_start":"465'7174","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":3599,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974232,"num_objects":2854,"num_object_clones":0,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":1730,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11914,"num_write_kb":472513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2492,"num_bytes_recovered":163315712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1730},{"shards":"4,6","objects":-915}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1178,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:30.154635+0000","last_change":"2024-03-26T21:48:32.957198+0000","last_active":"2024-03-26T21:50:30.154635+0000","last_peered":"2024-03-26T21:50:30.154635+0000","last_clean":"2024-03-26T21:50:30.154635+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:50:30.154635+0000","last_undegraded":"2024-03-26T21:50:30.154635+0000","last_fullsized":"2024-03-26T21:50:30.154635+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:48:32.957059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.957059+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":1009,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261019+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:50:40.261019+0000","last_peered":"2024-03-26T21:50:40.261019+0000","last_clean":"2024-03-26T21:50:40.261019+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:50:40.261019+0000","last_undegraded":"2024-03-26T21:50:40.261019+0000","last_fullsized":"2024-03-26T21:50:40.261019+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":947,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272857+0000","last_change":"2024-03-26T21:49:27.705867+0000","last_active":"2024-03-26T21:50:40.272857+0000","last_peered":"2024-03-26T21:50:40.272857+0000","last_clean":"2024-03-26T21:50:40.272857+0000","last_became_active":"2024-03-26T21:48:58.979760+0000","last_became_peered":"2024-03-26T21:48:58.979760+0000","last_unstale":"2024-03-26T21:50:40.272857+0000","last_undegraded":"2024-03-26T21:50:40.272857+0000","last_fullsized":"2024-03-26T21:50:40.272857+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:27.705733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:49:27.705733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:44.845395+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":919,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.113461+0000","last_change":"2024-03-26T21:48:58.973098+0000","last_active":"2024-03-26T21:50:25.113461+0000","last_peered":"2024-03-26T21:50:25.113461+0000","last_clean":"2024-03-26T21:50:25.113461+0000","last_became_active":"2024-03-26T21:48:58.972760+0000","last_became_peered":"2024-03-26T21:48:58.972760+0000","last_unstale":"2024-03-26T21:50:25.113461+0000","last_undegraded":"2024-03-26T21:50:25.113461+0000","last_fullsized":"2024-03-26T21:50:25.113461+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"486'12371","reported_seq":24075,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.376751+0000","last_change":"2024-03-26T21:50:16.112541+0000","last_active":"2024-03-26T21:50:43.376751+0000","last_peered":"2024-03-26T21:50:43.376751+0000","last_clean":"2024-03-26T21:50:43.376751+0000","last_became_active":"2024-03-26T21:50:16.057916+0000","last_became_peered":"2024-03-26T21:50:16.057916+0000","last_unstale":"2024-03-26T21:50:43.376751+0000","last_undegraded":"2024-03-26T21:50:43.376751+0000","last_fullsized":"2024-03-26T21:50:43.376751+0000","mapping_epoch":481,"log_start":"468'8056","ondisk_log_start":"468'8056","created":433,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180027392,"num_objects":2747,"num_object_clones":0,"num_object_copies":8241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12371,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5783,"num_bytes_recovered":378929152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114961+0000","last_change":"2024-03-26T21:49:29.979212+0000","last_active":"2024-03-26T21:50:25.114961+0000","last_peered":"2024-03-26T21:50:25.114961+0000","last_clean":"2024-03-26T21:50:25.114961+0000","last_became_active":"2024-03-26T21:49:29.978839+0000","last_became_peered":"2024-03-26T21:49:29.978839+0000","last_unstale":"2024-03-26T21:50:25.114961+0000","last_undegraded":"2024-03-26T21:50:25.114961+0000","last_fullsized":"2024-03-26T21:50:25.114961+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":984,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115194+0000","last_change":"2024-03-26T21:48:35.997876+0000","last_active":"2024-03-26T21:50:25.115194+0000","last_peered":"2024-03-26T21:50:25.115194+0000","last_clean":"2024-03-26T21:50:25.115194+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:50:25.115194+0000","last_undegraded":"2024-03-26T21:50:25.115194+0000","last_fullsized":"2024-03-26T21:50:25.115194+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:48:35.997722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.997722+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":945,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262806+0000","last_change":"2024-03-26T21:48:58.981784+0000","last_active":"2024-03-26T21:50:40.262806+0000","last_peered":"2024-03-26T21:50:40.262806+0000","last_clean":"2024-03-26T21:50:40.262806+0000","last_became_active":"2024-03-26T21:48:58.981168+0000","last_became_peered":"2024-03-26T21:48:58.981168+0000","last_unstale":"2024-03-26T21:50:40.262806+0000","last_undegraded":"2024-03-26T21:50:40.262806+0000","last_fullsized":"2024-03-26T21:50:40.262806+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":925,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261653+0000","last_change":"2024-03-26T21:48:58.983664+0000","last_active":"2024-03-26T21:50:40.261653+0000","last_peered":"2024-03-26T21:50:40.261653+0000","last_clean":"2024-03-26T21:50:40.261653+0000","last_became_active":"2024-03-26T21:48:58.983312+0000","last_became_peered":"2024-03-26T21:48:58.983312+0000","last_unstale":"2024-03-26T21:50:40.261653+0000","last_undegraded":"2024-03-26T21:50:40.261653+0000","last_fullsized":"2024-03-26T21:50:40.261653+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.809302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"486'12226","reported_seq":12343,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.604678+0000","last_change":"2024-03-26T21:47:39.047854+0000","last_active":"2024-03-26T21:50:43.604678+0000","last_peered":"2024-03-26T21:50:43.604678+0000","last_clean":"2024-03-26T21:50:43.604678+0000","last_became_active":"2024-03-26T21:47:39.047314+0000","last_became_peered":"2024-03-26T21:47:39.047314+0000","last_unstale":"2024-03-26T21:50:43.604678+0000","last_undegraded":"2024-03-26T21:50:43.604678+0000","last_fullsized":"2024-03-26T21:50:43.604678+0000","mapping_epoch":433,"log_start":"466'7618","ondisk_log_start":"466'7618","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12226,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":921,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273781+0000","last_change":"2024-03-26T21:49:30.726913+0000","last_active":"2024-03-26T21:50:40.273781+0000","last_peered":"2024-03-26T21:50:40.273781+0000","last_clean":"2024-03-26T21:50:40.273781+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:50:40.273781+0000","last_undegraded":"2024-03-26T21:50:40.273781+0000","last_fullsized":"2024-03-26T21:50:40.273781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:30.726780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:30.726780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.940297+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":1003,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.117312+0000","last_change":"2024-03-26T21:48:59.007492+0000","last_active":"2024-03-26T21:50:25.117312+0000","last_peered":"2024-03-26T21:50:25.117312+0000","last_clean":"2024-03-26T21:50:25.117312+0000","last_became_active":"2024-03-26T21:48:58.986240+0000","last_became_peered":"2024-03-26T21:48:58.986240+0000","last_unstale":"2024-03-26T21:50:25.117312+0000","last_undegraded":"2024-03-26T21:50:25.117312+0000","last_fullsized":"2024-03-26T21:50:25.117312+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2096,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:41.715377+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:50:41.715377+0000","last_peered":"2024-03-26T21:50:41.715377+0000","last_clean":"2024-03-26T21:50:41.715377+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:50:41.715377+0000","last_undegraded":"2024-03-26T21:50:41.715377+0000","last_fullsized":"2024-03-26T21:50:41.715377+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1207,"num_read_kb":1202,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":946,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114426+0000","last_change":"2024-03-26T21:48:41.039131+0000","last_active":"2024-03-26T21:50:25.114426+0000","last_peered":"2024-03-26T21:50:25.114426+0000","last_clean":"2024-03-26T21:50:25.114426+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:50:25.114426+0000","last_undegraded":"2024-03-26T21:50:25.114426+0000","last_fullsized":"2024-03-26T21:50:25.114426+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.035559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.035559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":931,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261996+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:50:40.261996+0000","last_peered":"2024-03-26T21:50:40.261996+0000","last_clean":"2024-03-26T21:50:40.261996+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:50:40.261996+0000","last_undegraded":"2024-03-26T21:50:40.261996+0000","last_fullsized":"2024-03-26T21:50:40.261996+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"486'12471","reported_seq":12586,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.587614+0000","last_change":"2024-03-26T21:47:39.047378+0000","last_active":"2024-03-26T21:50:43.587614+0000","last_peered":"2024-03-26T21:50:43.587614+0000","last_clean":"2024-03-26T21:50:43.587614+0000","last_became_active":"2024-03-26T21:47:39.047073+0000","last_became_peered":"2024-03-26T21:47:39.047073+0000","last_unstale":"2024-03-26T21:50:43.587614+0000","last_undegraded":"2024-03-26T21:50:43.587614+0000","last_fullsized":"2024-03-26T21:50:43.587614+0000","mapping_epoch":433,"log_start":"466'7874","ondisk_log_start":"466'7874","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170721280,"num_objects":2605,"num_object_clones":0,"num_object_copies":7815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":482432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":857,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266523+0000","last_change":"2024-03-26T21:48:58.992875+0000","last_active":"2024-03-26T21:50:40.266523+0000","last_peered":"2024-03-26T21:50:40.266523+0000","last_clean":"2024-03-26T21:50:40.266523+0000","last_became_active":"2024-03-26T21:48:58.992117+0000","last_became_peered":"2024-03-26T21:48:58.992117+0000","last_unstale":"2024-03-26T21:50:40.266523+0000","last_undegraded":"2024-03-26T21:50:40.266523+0000","last_fullsized":"2024-03-26T21:50:40.266523+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"468'55","reported_seq":1307,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.271898+0000","last_change":"2024-03-26T21:49:29.985516+0000","last_active":"2024-03-26T21:50:40.271898+0000","last_peered":"2024-03-26T21:50:40.271898+0000","last_clean":"2024-03-26T21:50:40.271898+0000","last_became_active":"2024-03-26T21:49:29.985117+0000","last_became_peered":"2024-03-26T21:49:29.985117+0000","last_unstale":"2024-03-26T21:50:40.271898+0000","last_undegraded":"2024-03-26T21:50:40.271898+0000","last_fullsized":"2024-03-26T21:50:40.271898+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.276458+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:50:40.276458+0000","last_peered":"2024-03-26T21:50:40.276458+0000","last_clean":"2024-03-26T21:50:40.276458+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:50:40.276458+0000","last_undegraded":"2024-03-26T21:50:40.276458+0000","last_fullsized":"2024-03-26T21:50:40.276458+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1120,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266078+0000","last_change":"2024-03-26T21:48:27.821566+0000","last_active":"2024-03-26T21:50:40.266078+0000","last_peered":"2024-03-26T21:50:40.266078+0000","last_clean":"2024-03-26T21:50:40.266078+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:50:40.266078+0000","last_undegraded":"2024-03-26T21:50:40.266078+0000","last_fullsized":"2024-03-26T21:50:40.266078+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:48:27.821430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.821430+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":942,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273372+0000","last_change":"2024-03-26T21:49:29.780022+0000","last_active":"2024-03-26T21:50:40.273372+0000","last_peered":"2024-03-26T21:50:40.273372+0000","last_clean":"2024-03-26T21:50:40.273372+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:50:40.273372+0000","last_undegraded":"2024-03-26T21:50:40.273372+0000","last_fullsized":"2024-03-26T21:50:40.273372+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:29.779883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:29.779883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:46.086766+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"486'12447","reported_seq":12564,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.585782+0000","last_change":"2024-03-26T21:47:39.047146+0000","last_active":"2024-03-26T21:50:43.585782+0000","last_peered":"2024-03-26T21:50:43.585782+0000","last_clean":"2024-03-26T21:50:43.585782+0000","last_became_active":"2024-03-26T21:47:39.046837+0000","last_became_peered":"2024-03-26T21:47:39.046837+0000","last_unstale":"2024-03-26T21:50:43.585782+0000","last_undegraded":"2024-03-26T21:50:43.585782+0000","last_fullsized":"2024-03-26T21:50:43.585782+0000","mapping_epoch":433,"log_start":"466'7834","ondisk_log_start":"466'7834","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":536,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273017+0000","last_change":"2024-03-26T21:49:19.802374+0000","last_active":"2024-03-26T21:50:40.273017+0000","last_peered":"2024-03-26T21:50:40.273017+0000","last_clean":"2024-03-26T21:50:40.273017+0000","last_became_active":"2024-03-26T21:48:58.985930+0000","last_became_peered":"2024-03-26T21:48:58.985930+0000","last_unstale":"2024-03-26T21:50:40.273017+0000","last_undegraded":"2024-03-26T21:50:40.273017+0000","last_fullsized":"2024-03-26T21:50:40.273017+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:19.802252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:49:19.802252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:42.836619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":979,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261237+0000","last_change":"2024-03-26T21:48:35.767934+0000","last_active":"2024-03-26T21:50:40.261237+0000","last_peered":"2024-03-26T21:50:40.261237+0000","last_clean":"2024-03-26T21:50:40.261237+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:50:40.261237+0000","last_undegraded":"2024-03-26T21:50:40.261237+0000","last_fullsized":"2024-03-26T21:50:40.261237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:48:35.767034+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.767034+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":746,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261188+0000","last_change":"2024-03-26T21:48:58.978047+0000","last_active":"2024-03-26T21:50:40.261188+0000","last_peered":"2024-03-26T21:50:40.261188+0000","last_clean":"2024-03-26T21:50:40.261188+0000","last_became_active":"2024-03-26T21:48:58.976535+0000","last_became_peered":"2024-03-26T21:48:58.976535+0000","last_unstale":"2024-03-26T21:50:40.261188+0000","last_undegraded":"2024-03-26T21:50:40.261188+0000","last_fullsized":"2024-03-26T21:50:40.261188+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":946,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261912+0000","last_change":"2024-03-26T21:48:58.982624+0000","last_active":"2024-03-26T21:50:40.261912+0000","last_peered":"2024-03-26T21:50:40.261912+0000","last_clean":"2024-03-26T21:50:40.261912+0000","last_became_active":"2024-03-26T21:48:58.982052+0000","last_became_peered":"2024-03-26T21:48:58.982052+0000","last_unstale":"2024-03-26T21:50:40.261912+0000","last_undegraded":"2024-03-26T21:50:40.261912+0000","last_fullsized":"2024-03-26T21:50:40.261912+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":839,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266703+0000","last_change":"2024-03-26T21:48:31.817091+0000","last_active":"2024-03-26T21:50:40.266703+0000","last_peered":"2024-03-26T21:50:40.266703+0000","last_clean":"2024-03-26T21:50:40.266703+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:50:40.266703+0000","last_undegraded":"2024-03-26T21:50:40.266703+0000","last_fullsized":"2024-03-26T21:50:40.266703+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:31.817002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.817002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"486'12048","reported_seq":12165,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.569373+0000","last_change":"2024-03-26T21:47:39.051677+0000","last_active":"2024-03-26T21:50:43.569373+0000","last_peered":"2024-03-26T21:50:43.569373+0000","last_clean":"2024-03-26T21:50:43.569373+0000","last_became_active":"2024-03-26T21:47:39.051279+0000","last_became_peered":"2024-03-26T21:47:39.051279+0000","last_unstale":"2024-03-26T21:50:43.569373+0000","last_undegraded":"2024-03-26T21:50:43.569373+0000","last_fullsized":"2024-03-26T21:50:43.569373+0000","mapping_epoch":433,"log_start":"465'7404","ondisk_log_start":"465'7404","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174981120,"num_objects":2670,"num_object_clones":0,"num_object_copies":8010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":470976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":926,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115365+0000","last_change":"2024-03-26T21:48:32.000228+0000","last_active":"2024-03-26T21:50:25.115365+0000","last_peered":"2024-03-26T21:50:25.115365+0000","last_clean":"2024-03-26T21:50:25.115365+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:50:25.115365+0000","last_undegraded":"2024-03-26T21:50:25.115365+0000","last_fullsized":"2024-03-26T21:50:25.115365+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.000079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"462'56","reported_seq":1315,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272265+0000","last_change":"2024-03-26T21:49:57.671639+0000","last_active":"2024-03-26T21:50:40.272265+0000","last_peered":"2024-03-26T21:50:40.272265+0000","last_clean":"2024-03-26T21:50:40.272265+0000","last_became_active":"2024-03-26T21:48:58.981458+0000","last_became_peered":"2024-03-26T21:48:58.981458+0000","last_unstale":"2024-03-26T21:50:40.272265+0000","last_undegraded":"2024-03-26T21:50:40.272265+0000","last_fullsized":"2024-03-26T21:50:40.272265+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'56","last_scrub_stamp":"2024-03-26T21:49:57.671485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:49:57.671485+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:12.579557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":59,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262007+0000","last_change":"2024-03-26T21:48:58.976086+0000","last_active":"2024-03-26T21:50:40.262007+0000","last_peered":"2024-03-26T21:50:40.262007+0000","last_clean":"2024-03-26T21:50:40.262007+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:50:40.262007+0000","last_undegraded":"2024-03-26T21:50:40.262007+0000","last_fullsized":"2024-03-26T21:50:40.262007+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":939,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272138+0000","last_change":"2024-03-26T21:48:43.783835+0000","last_active":"2024-03-26T21:50:40.272138+0000","last_peered":"2024-03-26T21:50:40.272138+0000","last_clean":"2024-03-26T21:50:40.272138+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:50:40.272138+0000","last_undegraded":"2024-03-26T21:50:40.272138+0000","last_fullsized":"2024-03-26T21:50:40.272138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:48:43.783585+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:42.836590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.263362+0000","last_change":"2024-03-26T21:48:29.771118+0000","last_active":"2024-03-26T21:50:40.263362+0000","last_peered":"2024-03-26T21:50:40.263362+0000","last_clean":"2024-03-26T21:50:40.263362+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:50:40.263362+0000","last_undegraded":"2024-03-26T21:50:40.263362+0000","last_fullsized":"2024-03-26T21:50:40.263362+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.770950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.770950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"486'12331","reported_seq":25786,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.568098+0000","last_change":"2024-03-26T21:49:57.447616+0000","last_active":"2024-03-26T21:50:43.568098+0000","last_peered":"2024-03-26T21:50:43.568098+0000","last_clean":"2024-03-26T21:50:43.568098+0000","last_became_active":"2024-03-26T21:49:57.435919+0000","last_became_peered":"2024-03-26T21:49:57.435919+0000","last_unstale":"2024-03-26T21:50:43.568098+0000","last_undegraded":"2024-03-26T21:50:43.568098+0000","last_fullsized":"2024-03-26T21:50:43.568098+0000","mapping_epoch":476,"log_start":"466'7624","ondisk_log_start":"466'7624","created":433,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173735936,"num_objects":2651,"num_object_clones":0,"num_object_copies":7953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12331,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6662,"num_bytes_recovered":436600832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":967,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115338+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:50:25.115338+0000","last_peered":"2024-03-26T21:50:25.115338+0000","last_clean":"2024-03-26T21:50:25.115338+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:50:25.115338+0000","last_undegraded":"2024-03-26T21:50:25.115338+0000","last_fullsized":"2024-03-26T21:50:25.115338+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":948,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272578+0000","last_change":"2024-03-26T21:49:31.653285+0000","last_active":"2024-03-26T21:50:40.272578+0000","last_peered":"2024-03-26T21:50:40.272578+0000","last_clean":"2024-03-26T21:50:40.272578+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:50:40.272578+0000","last_undegraded":"2024-03-26T21:50:40.272578+0000","last_fullsized":"2024-03-26T21:50:40.272578+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:31.653167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:49:31.653167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:59.137490+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":744,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262230+0000","last_change":"2024-03-26T21:48:58.968424+0000","last_active":"2024-03-26T21:50:40.262230+0000","last_peered":"2024-03-26T21:50:40.262230+0000","last_clean":"2024-03-26T21:50:40.262230+0000","last_became_active":"2024-03-26T21:48:58.968010+0000","last_became_peered":"2024-03-26T21:48:58.968010+0000","last_unstale":"2024-03-26T21:50:40.262230+0000","last_undegraded":"2024-03-26T21:50:40.262230+0000","last_fullsized":"2024-03-26T21:50:40.262230+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":908,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114606+0000","last_change":"2024-03-26T21:48:58.983049+0000","last_active":"2024-03-26T21:50:25.114606+0000","last_peered":"2024-03-26T21:50:25.114606+0000","last_clean":"2024-03-26T21:50:25.114606+0000","last_became_active":"2024-03-26T21:48:58.982744+0000","last_became_peered":"2024-03-26T21:48:58.982744+0000","last_unstale":"2024-03-26T21:50:25.114606+0000","last_undegraded":"2024-03-26T21:50:25.114606+0000","last_fullsized":"2024-03-26T21:50:25.114606+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"486'11820","reported_seq":11937,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:43.492820+0000","last_change":"2024-03-26T21:47:39.046775+0000","last_active":"2024-03-26T21:50:43.492820+0000","last_peered":"2024-03-26T21:50:43.492820+0000","last_clean":"2024-03-26T21:50:43.492820+0000","last_became_active":"2024-03-26T21:47:39.046369+0000","last_became_peered":"2024-03-26T21:47:39.046369+0000","last_unstale":"2024-03-26T21:50:43.492820+0000","last_undegraded":"2024-03-26T21:50:43.492820+0000","last_fullsized":"2024-03-26T21:50:43.492820+0000","mapping_epoch":433,"log_start":"465'7157","ondisk_log_start":"465'7157","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1009,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260825+0000","last_change":"2024-03-26T21:49:29.981307+0000","last_active":"2024-03-26T21:50:40.260825+0000","last_peered":"2024-03-26T21:50:40.260825+0000","last_clean":"2024-03-26T21:50:40.260825+0000","last_became_active":"2024-03-26T21:49:29.980914+0000","last_became_peered":"2024-03-26T21:49:29.980914+0000","last_unstale":"2024-03-26T21:50:40.260825+0000","last_undegraded":"2024-03-26T21:50:40.260825+0000","last_fullsized":"2024-03-26T21:50:40.260825+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":874,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266902+0000","last_change":"2024-03-26T21:48:44.612681+0000","last_active":"2024-03-26T21:50:40.266902+0000","last_peered":"2024-03-26T21:50:40.266902+0000","last_clean":"2024-03-26T21:50:40.266902+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:50:40.266902+0000","last_undegraded":"2024-03-26T21:50:40.266902+0000","last_fullsized":"2024-03-26T21:50:40.266902+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.612470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114721+0000","last_change":"2024-03-26T21:48:58.986135+0000","last_active":"2024-03-26T21:50:25.114721+0000","last_peered":"2024-03-26T21:50:25.114721+0000","last_clean":"2024-03-26T21:50:25.114721+0000","last_became_active":"2024-03-26T21:48:58.985788+0000","last_became_peered":"2024-03-26T21:48:58.985788+0000","last_unstale":"2024-03-26T21:50:25.114721+0000","last_undegraded":"2024-03-26T21:50:25.114721+0000","last_fullsized":"2024-03-26T21:50:25.114721+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":845,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.265426+0000","last_change":"2024-03-26T21:48:58.984676+0000","last_active":"2024-03-26T21:50:40.265426+0000","last_peered":"2024-03-26T21:50:40.265426+0000","last_clean":"2024-03-26T21:50:40.265426+0000","last_became_active":"2024-03-26T21:48:58.983980+0000","last_became_peered":"2024-03-26T21:48:58.983980+0000","last_unstale":"2024-03-26T21:50:40.265426+0000","last_undegraded":"2024-03-26T21:50:40.265426+0000","last_fullsized":"2024-03-26T21:50:40.265426+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"485'11718","reported_seq":25353,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:39.058488+0000","last_change":"2024-03-26T21:49:45.257806+0000","last_active":"2024-03-26T21:50:39.058488+0000","last_peered":"2024-03-26T21:50:39.058488+0000","last_clean":"2024-03-26T21:50:39.058488+0000","last_became_active":"2024-03-26T21:49:45.247163+0000","last_became_peered":"2024-03-26T21:49:45.247163+0000","last_unstale":"2024-03-26T21:50:39.058488+0000","last_undegraded":"2024-03-26T21:50:39.058488+0000","last_fullsized":"2024-03-26T21:50:39.058488+0000","mapping_epoch":471,"log_start":"464'6911","ondisk_log_start":"464'6911","created":433,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:47:38.017744+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194772992,"num_objects":2972,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11718,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6755,"num_bytes_recovered":442695680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.113804+0000","last_change":"2024-03-26T21:48:58.977207+0000","last_active":"2024-03-26T21:50:25.113804+0000","last_peered":"2024-03-26T21:50:25.113804+0000","last_clean":"2024-03-26T21:50:25.113804+0000","last_became_active":"2024-03-26T21:48:58.976826+0000","last_became_peered":"2024-03-26T21:48:58.976826+0000","last_unstale":"2024-03-26T21:50:25.113804+0000","last_undegraded":"2024-03-26T21:50:25.113804+0000","last_fullsized":"2024-03-26T21:50:25.113804+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":682,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262686+0000","last_change":"2024-03-26T21:48:36.706426+0000","last_active":"2024-03-26T21:50:40.262686+0000","last_peered":"2024-03-26T21:50:40.262686+0000","last_clean":"2024-03-26T21:50:40.262686+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:50:40.262686+0000","last_undegraded":"2024-03-26T21:50:40.262686+0000","last_fullsized":"2024-03-26T21:50:40.262686+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:48:36.706278+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":904,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.267201+0000","last_change":"2024-03-26T21:48:35.727970+0000","last_active":"2024-03-26T21:50:40.267201+0000","last_peered":"2024-03-26T21:50:40.267201+0000","last_clean":"2024-03-26T21:50:40.267201+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:50:40.267201+0000","last_undegraded":"2024-03-26T21:50:40.267201+0000","last_fullsized":"2024-03-26T21:50:40.267201+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.727841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262764+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:50:40.262764+0000","last_peered":"2024-03-26T21:50:40.262764+0000","last_clean":"2024-03-26T21:50:40.262764+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:50:40.262764+0000","last_undegraded":"2024-03-26T21:50:40.262764+0000","last_fullsized":"2024-03-26T21:50:40.262764+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"486'12102","reported_seq":12300,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:42.138578+0000","last_change":"2024-03-26T21:49:25.777937+0000","last_active":"2024-03-26T21:50:42.138578+0000","last_peered":"2024-03-26T21:50:42.138578+0000","last_clean":"2024-03-26T21:50:42.138578+0000","last_became_active":"2024-03-26T21:47:39.051349+0000","last_became_peered":"2024-03-26T21:47:39.051349+0000","last_unstale":"2024-03-26T21:50:42.138578+0000","last_undegraded":"2024-03-26T21:50:42.138578+0000","last_fullsized":"2024-03-26T21:50:42.138578+0000","mapping_epoch":433,"log_start":"464'6837","ondisk_log_start":"464'6837","created":433,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'7654","last_scrub_stamp":"2024-03-26T21:49:25.777551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:38.017744+0000","last_clean_scrub_stamp":"2024-03-26T21:49:25.777551+0000","objects_scrubbed":7294,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:47.024151+0000","scrub_duration":2010,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1125,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.265643+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:50:40.265643+0000","last_peered":"2024-03-26T21:50:40.265643+0000","last_clean":"2024-03-26T21:50:40.265643+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:50:40.265643+0000","last_undegraded":"2024-03-26T21:50:40.265643+0000","last_fullsized":"2024-03-26T21:50:40.265643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":1002,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.271914+0000","last_change":"2024-03-26T21:49:38.645618+0000","last_active":"2024-03-26T21:50:40.271914+0000","last_peered":"2024-03-26T21:50:40.271914+0000","last_clean":"2024-03-26T21:50:40.271914+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:50:40.271914+0000","last_undegraded":"2024-03-26T21:50:40.271914+0000","last_fullsized":"2024-03-26T21:50:40.271914+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:49:38.645456+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:57.832210+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":913,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.267135+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:50:40.267135+0000","last_peered":"2024-03-26T21:50:40.267135+0000","last_clean":"2024-03-26T21:50:40.267135+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:50:40.267135+0000","last_undegraded":"2024-03-26T21:50:40.267135+0000","last_fullsized":"2024-03-26T21:50:40.267135+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":917,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273410+0000","last_change":"2024-03-26T21:49:35.727593+0000","last_active":"2024-03-26T21:50:40.273410+0000","last_peered":"2024-03-26T21:50:40.273410+0000","last_clean":"2024-03-26T21:50:40.273410+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:50:40.273410+0000","last_undegraded":"2024-03-26T21:50:40.273410+0000","last_fullsized":"2024-03-26T21:50:40.273410+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:35.727500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:35.727500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:52.939494+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":296,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260918+0000","last_change":"2024-03-26T21:48:59.002050+0000","last_active":"2024-03-26T21:50:40.260918+0000","last_peered":"2024-03-26T21:50:40.260918+0000","last_clean":"2024-03-26T21:50:40.260918+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:50:40.260918+0000","last_undegraded":"2024-03-26T21:50:40.260918+0000","last_fullsized":"2024-03-26T21:50:40.260918+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114021+0000","last_change":"2024-03-26T21:49:29.977391+0000","last_active":"2024-03-26T21:50:25.114021+0000","last_peered":"2024-03-26T21:50:25.114021+0000","last_clean":"2024-03-26T21:50:25.114021+0000","last_became_active":"2024-03-26T21:49:29.976836+0000","last_became_peered":"2024-03-26T21:49:29.976836+0000","last_unstale":"2024-03-26T21:50:25.114021+0000","last_undegraded":"2024-03-26T21:50:25.114021+0000","last_fullsized":"2024-03-26T21:50:25.114021+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.291965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":932,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273171+0000","last_change":"2024-03-26T21:49:41.705007+0000","last_active":"2024-03-26T21:50:40.273171+0000","last_peered":"2024-03-26T21:50:40.273171+0000","last_clean":"2024-03-26T21:50:40.273171+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:50:40.273171+0000","last_undegraded":"2024-03-26T21:50:40.273171+0000","last_fullsized":"2024-03-26T21:50:40.273171+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:41.704871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:41.704871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:49.588465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":848,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266757+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:50:40.266757+0000","last_peered":"2024-03-26T21:50:40.266757+0000","last_clean":"2024-03-26T21:50:40.266757+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:50:40.266757+0000","last_undegraded":"2024-03-26T21:50:40.266757+0000","last_fullsized":"2024-03-26T21:50:40.266757+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":985,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.118991+0000","last_change":"2024-03-26T21:48:44.101005+0000","last_active":"2024-03-26T21:50:25.118991+0000","last_peered":"2024-03-26T21:50:25.118991+0000","last_clean":"2024-03-26T21:50:25.118991+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:50:25.118991+0000","last_undegraded":"2024-03-26T21:50:25.118991+0000","last_fullsized":"2024-03-26T21:50:25.118991+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:48:44.100867+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.100867+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":886,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.265880+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:50:40.265880+0000","last_peered":"2024-03-26T21:50:40.265880+0000","last_clean":"2024-03-26T21:50:40.265880+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:50:40.265880+0000","last_undegraded":"2024-03-26T21:50:40.265880+0000","last_fullsized":"2024-03-26T21:50:40.265880+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261069+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:50:40.261069+0000","last_peered":"2024-03-26T21:50:40.261069+0000","last_clean":"2024-03-26T21:50:40.261069+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:50:40.261069+0000","last_undegraded":"2024-03-26T21:50:40.261069+0000","last_fullsized":"2024-03-26T21:50:40.261069+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":545,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261930+0000","last_change":"2024-03-26T21:48:58.984295+0000","last_active":"2024-03-26T21:50:40.261930+0000","last_peered":"2024-03-26T21:50:40.261930+0000","last_clean":"2024-03-26T21:50:40.261930+0000","last_became_active":"2024-03-26T21:48:58.983139+0000","last_became_peered":"2024-03-26T21:48:58.983139+0000","last_unstale":"2024-03-26T21:50:40.261930+0000","last_undegraded":"2024-03-26T21:50:40.261930+0000","last_fullsized":"2024-03-26T21:50:40.261930+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":1001,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273051+0000","last_change":"2024-03-26T21:50:00.738089+0000","last_active":"2024-03-26T21:50:40.273051+0000","last_peered":"2024-03-26T21:50:40.273051+0000","last_clean":"2024-03-26T21:50:40.273051+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:50:40.273051+0000","last_undegraded":"2024-03-26T21:50:40.273051+0000","last_fullsized":"2024-03-26T21:50:40.273051+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:50:00.737954+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:50:00.737954+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:01.952910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":923,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.114868+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:50:25.114868+0000","last_peered":"2024-03-26T21:50:25.114868+0000","last_clean":"2024-03-26T21:50:25.114868+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:50:25.114868+0000","last_undegraded":"2024-03-26T21:50:25.114868+0000","last_fullsized":"2024-03-26T21:50:25.114868+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"462'14","reported_seq":351,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260620+0000","last_change":"2024-03-26T21:48:58.993774+0000","last_active":"2024-03-26T21:50:40.260620+0000","last_peered":"2024-03-26T21:50:40.260620+0000","last_clean":"2024-03-26T21:50:40.260620+0000","last_became_active":"2024-03-26T21:48:58.989325+0000","last_became_peered":"2024-03-26T21:48:58.989325+0000","last_unstale":"2024-03-26T21:50:40.260620+0000","last_undegraded":"2024-03-26T21:50:40.260620+0000","last_fullsized":"2024-03-26T21:50:40.260620+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13","last_scrub_stamp":"2024-03-26T21:48:33.859117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.859117+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":615,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.263528+0000","last_change":"2024-03-26T21:48:58.970848+0000","last_active":"2024-03-26T21:50:40.263528+0000","last_peered":"2024-03-26T21:50:40.263528+0000","last_clean":"2024-03-26T21:50:40.263528+0000","last_became_active":"2024-03-26T21:48:58.970509+0000","last_became_peered":"2024-03-26T21:48:58.970509+0000","last_unstale":"2024-03-26T21:50:40.263528+0000","last_undegraded":"2024-03-26T21:50:40.263528+0000","last_fullsized":"2024-03-26T21:50:40.263528+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":292,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260366+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:50:40.260366+0000","last_peered":"2024-03-26T21:50:40.260366+0000","last_clean":"2024-03-26T21:50:40.260366+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:50:40.260366+0000","last_undegraded":"2024-03-26T21:50:40.260366+0000","last_fullsized":"2024-03-26T21:50:40.260366+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":865,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266576+0000","last_change":"2024-03-26T21:48:58.987967+0000","last_active":"2024-03-26T21:50:40.266576+0000","last_peered":"2024-03-26T21:50:40.266576+0000","last_clean":"2024-03-26T21:50:40.266576+0000","last_became_active":"2024-03-26T21:48:58.987113+0000","last_became_peered":"2024-03-26T21:48:58.987113+0000","last_unstale":"2024-03-26T21:50:40.266576+0000","last_undegraded":"2024-03-26T21:50:40.266576+0000","last_fullsized":"2024-03-26T21:50:40.266576+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1195,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272767+0000","last_change":"2024-03-26T21:50:05.674011+0000","last_active":"2024-03-26T21:50:40.272767+0000","last_peered":"2024-03-26T21:50:40.272767+0000","last_clean":"2024-03-26T21:50:40.272767+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:50:40.272767+0000","last_undegraded":"2024-03-26T21:50:40.272767+0000","last_fullsized":"2024-03-26T21:50:40.272767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:50:05.673903+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:50:05.673903+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:21.698146+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":920,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.117235+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:50:25.117235+0000","last_peered":"2024-03-26T21:50:25.117235+0000","last_clean":"2024-03-26T21:50:25.117235+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:50:25.117235+0000","last_undegraded":"2024-03-26T21:50:25.117235+0000","last_fullsized":"2024-03-26T21:50:25.117235+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":651,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262343+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:50:40.262343+0000","last_peered":"2024-03-26T21:50:40.262343+0000","last_clean":"2024-03-26T21:50:40.262343+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:50:40.262343+0000","last_undegraded":"2024-03-26T21:50:40.262343+0000","last_fullsized":"2024-03-26T21:50:40.262343+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":619,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262086+0000","last_change":"2024-03-26T21:48:39.735268+0000","last_active":"2024-03-26T21:50:40.262086+0000","last_peered":"2024-03-26T21:50:40.262086+0000","last_clean":"2024-03-26T21:50:40.262086+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:50:40.262086+0000","last_undegraded":"2024-03-26T21:50:40.262086+0000","last_fullsized":"2024-03-26T21:50:40.262086+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:48:39.735146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":946,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.260928+0000","last_change":"2024-03-26T21:48:24.700288+0000","last_active":"2024-03-26T21:50:40.260928+0000","last_peered":"2024-03-26T21:50:40.260928+0000","last_clean":"2024-03-26T21:50:40.260928+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:50:40.260928+0000","last_undegraded":"2024-03-26T21:50:40.260928+0000","last_fullsized":"2024-03-26T21:50:40.260928+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:48:24.700160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261369+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:50:40.261369+0000","last_peered":"2024-03-26T21:50:40.261369+0000","last_clean":"2024-03-26T21:50:40.261369+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:50:40.261369+0000","last_undegraded":"2024-03-26T21:50:40.261369+0000","last_fullsized":"2024-03-26T21:50:40.261369+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":980,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261115+0000","last_change":"2024-03-26T21:48:59.005271+0000","last_active":"2024-03-26T21:50:40.261115+0000","last_peered":"2024-03-26T21:50:40.261115+0000","last_clean":"2024-03-26T21:50:40.261115+0000","last_became_active":"2024-03-26T21:48:58.979497+0000","last_became_peered":"2024-03-26T21:48:58.979497+0000","last_unstale":"2024-03-26T21:50:40.261115+0000","last_undegraded":"2024-03-26T21:50:40.261115+0000","last_fullsized":"2024-03-26T21:50:40.261115+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:30.679017+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.679017+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":863,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266804+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:50:40.266804+0000","last_peered":"2024-03-26T21:50:40.266804+0000","last_clean":"2024-03-26T21:50:40.266804+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:50:40.266804+0000","last_undegraded":"2024-03-26T21:50:40.266804+0000","last_fullsized":"2024-03-26T21:50:40.266804+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262184+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:50:40.262184+0000","last_peered":"2024-03-26T21:50:40.262184+0000","last_clean":"2024-03-26T21:50:40.262184+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:50:40.262184+0000","last_undegraded":"2024-03-26T21:50:40.262184+0000","last_fullsized":"2024-03-26T21:50:40.262184+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":900,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.113923+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:50:25.113923+0000","last_peered":"2024-03-26T21:50:25.113923+0000","last_clean":"2024-03-26T21:50:25.113923+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:50:25.113923+0000","last_undegraded":"2024-03-26T21:50:25.113923+0000","last_fullsized":"2024-03-26T21:50:25.113923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":297,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261616+0000","last_change":"2024-03-26T21:48:38.975837+0000","last_active":"2024-03-26T21:50:40.261616+0000","last_peered":"2024-03-26T21:50:40.261616+0000","last_clean":"2024-03-26T21:50:40.261616+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:50:40.261616+0000","last_undegraded":"2024-03-26T21:50:40.261616+0000","last_fullsized":"2024-03-26T21:50:40.261616+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:38.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:38.975710+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":937,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272878+0000","last_change":"2024-03-26T21:49:55.627083+0000","last_active":"2024-03-26T21:50:40.272878+0000","last_peered":"2024-03-26T21:50:40.272878+0000","last_clean":"2024-03-26T21:50:40.272878+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:50:40.272878+0000","last_undegraded":"2024-03-26T21:50:40.272878+0000","last_fullsized":"2024-03-26T21:50:40.272878+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:55.626947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:49:55.626947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:07.640931+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":841,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266030+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:50:40.266030+0000","last_peered":"2024-03-26T21:50:40.266030+0000","last_clean":"2024-03-26T21:50:40.266030+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:50:40.266030+0000","last_undegraded":"2024-03-26T21:50:40.266030+0000","last_fullsized":"2024-03-26T21:50:40.266030+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":273,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262681+0000","last_change":"2024-03-26T21:48:37.104233+0000","last_active":"2024-03-26T21:50:40.262681+0000","last_peered":"2024-03-26T21:50:40.262681+0000","last_clean":"2024-03-26T21:50:40.262681+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:50:40.262681+0000","last_undegraded":"2024-03-26T21:50:40.262681+0000","last_fullsized":"2024-03-26T21:50:40.262681+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.104108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.104108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":285,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262002+0000","last_change":"2024-03-26T21:48:42.913651+0000","last_active":"2024-03-26T21:50:40.262002+0000","last_peered":"2024-03-26T21:50:40.262002+0000","last_clean":"2024-03-26T21:50:40.262002+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:50:40.262002+0000","last_undegraded":"2024-03-26T21:50:40.262002+0000","last_fullsized":"2024-03-26T21:50:40.262002+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:48:42.913438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:42.913438+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":924,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115800+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:50:25.115800+0000","last_peered":"2024-03-26T21:50:25.115800+0000","last_clean":"2024-03-26T21:50:25.115800+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:50:25.115800+0000","last_undegraded":"2024-03-26T21:50:25.115800+0000","last_fullsized":"2024-03-26T21:50:25.115800+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":537,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262386+0000","last_change":"2024-03-26T21:49:29.976926+0000","last_active":"2024-03-26T21:50:40.262386+0000","last_peered":"2024-03-26T21:50:40.262386+0000","last_clean":"2024-03-26T21:50:40.262386+0000","last_became_active":"2024-03-26T21:49:29.976388+0000","last_became_peered":"2024-03-26T21:49:29.976388+0000","last_unstale":"2024-03-26T21:50:40.262386+0000","last_undegraded":"2024-03-26T21:50:40.262386+0000","last_fullsized":"2024-03-26T21:50:40.262386+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":855,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.268573+0000","last_change":"2024-03-26T21:48:58.980520+0000","last_active":"2024-03-26T21:50:40.268573+0000","last_peered":"2024-03-26T21:50:40.268573+0000","last_clean":"2024-03-26T21:50:40.268573+0000","last_became_active":"2024-03-26T21:48:58.980151+0000","last_became_peered":"2024-03-26T21:48:58.980151+0000","last_unstale":"2024-03-26T21:50:40.268573+0000","last_undegraded":"2024-03-26T21:50:40.268573+0000","last_fullsized":"2024-03-26T21:50:40.268573+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":976,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.273663+0000","last_change":"2024-03-26T21:49:50.705427+0000","last_active":"2024-03-26T21:50:40.273663+0000","last_peered":"2024-03-26T21:50:40.273663+0000","last_clean":"2024-03-26T21:50:40.273663+0000","last_became_active":"2024-03-26T21:49:29.980233+0000","last_became_peered":"2024-03-26T21:49:29.980233+0000","last_unstale":"2024-03-26T21:50:40.273663+0000","last_undegraded":"2024-03-26T21:50:40.273663+0000","last_fullsized":"2024-03-26T21:50:40.273663+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:49:50.705277+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:49:50.705277+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:06.503329+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261187+0000","last_change":"2024-03-26T21:48:26.761549+0000","last_active":"2024-03-26T21:50:40.261187+0000","last_peered":"2024-03-26T21:50:40.261187+0000","last_clean":"2024-03-26T21:50:40.261187+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:50:40.261187+0000","last_undegraded":"2024-03-26T21:50:40.261187+0000","last_fullsized":"2024-03-26T21:50:40.261187+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.760812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.760812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":920,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.123224+0000","last_change":"2024-03-26T21:48:58.986355+0000","last_active":"2024-03-26T21:50:25.123224+0000","last_peered":"2024-03-26T21:50:25.123224+0000","last_clean":"2024-03-26T21:50:25.123224+0000","last_became_active":"2024-03-26T21:48:58.985913+0000","last_became_peered":"2024-03-26T21:48:58.985913+0000","last_unstale":"2024-03-26T21:50:25.123224+0000","last_undegraded":"2024-03-26T21:50:25.123224+0000","last_fullsized":"2024-03-26T21:50:25.123224+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:48:45.073486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":938,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.283609+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:50:40.283609+0000","last_peered":"2024-03-26T21:50:40.283609+0000","last_clean":"2024-03-26T21:50:40.283609+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:50:40.283609+0000","last_undegraded":"2024-03-26T21:50:40.283609+0000","last_fullsized":"2024-03-26T21:50:40.283609+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1007,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.118083+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:50:25.118083+0000","last_peered":"2024-03-26T21:50:25.118083+0000","last_clean":"2024-03-26T21:50:25.118083+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:50:25.118083+0000","last_undegraded":"2024-03-26T21:50:25.118083+0000","last_fullsized":"2024-03-26T21:50:25.118083+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":973,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261804+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:50:40.261804+0000","last_peered":"2024-03-26T21:50:40.261804+0000","last_clean":"2024-03-26T21:50:40.261804+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:50:40.261804+0000","last_undegraded":"2024-03-26T21:50:40.261804+0000","last_fullsized":"2024-03-26T21:50:40.261804+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"468'15","reported_seq":507,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.263288+0000","last_change":"2024-03-26T21:49:29.976278+0000","last_active":"2024-03-26T21:50:40.263288+0000","last_peered":"2024-03-26T21:50:40.263288+0000","last_clean":"2024-03-26T21:50:40.263288+0000","last_became_active":"2024-03-26T21:49:29.975921+0000","last_became_peered":"2024-03-26T21:49:29.975921+0000","last_unstale":"2024-03-26T21:50:40.263288+0000","last_undegraded":"2024-03-26T21:50:40.263288+0000","last_fullsized":"2024-03-26T21:50:40.263288+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:48:32.327875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.327875+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":854,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.266414+0000","last_change":"2024-03-26T21:48:58.998136+0000","last_active":"2024-03-26T21:50:40.266414+0000","last_peered":"2024-03-26T21:50:40.266414+0000","last_clean":"2024-03-26T21:50:40.266414+0000","last_became_active":"2024-03-26T21:48:58.991425+0000","last_became_peered":"2024-03-26T21:48:58.991425+0000","last_unstale":"2024-03-26T21:50:40.266414+0000","last_undegraded":"2024-03-26T21:50:40.266414+0000","last_fullsized":"2024-03-26T21:50:40.266414+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":291,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261721+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:50:40.261721+0000","last_peered":"2024-03-26T21:50:40.261721+0000","last_clean":"2024-03-26T21:50:40.261721+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:50:40.261721+0000","last_undegraded":"2024-03-26T21:50:40.261721+0000","last_fullsized":"2024-03-26T21:50:40.261721+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":616,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261620+0000","last_change":"2024-03-26T21:48:58.973845+0000","last_active":"2024-03-26T21:50:40.261620+0000","last_peered":"2024-03-26T21:50:40.261620+0000","last_clean":"2024-03-26T21:50:40.261620+0000","last_became_active":"2024-03-26T21:48:58.973428+0000","last_became_peered":"2024-03-26T21:48:58.973428+0000","last_unstale":"2024-03-26T21:50:40.261620+0000","last_undegraded":"2024-03-26T21:50:40.261620+0000","last_fullsized":"2024-03-26T21:50:40.261620+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":640,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262242+0000","last_change":"2024-03-26T21:48:58.983537+0000","last_active":"2024-03-26T21:50:40.262242+0000","last_peered":"2024-03-26T21:50:40.262242+0000","last_clean":"2024-03-26T21:50:40.262242+0000","last_became_active":"2024-03-26T21:48:58.982850+0000","last_became_peered":"2024-03-26T21:48:58.982850+0000","last_unstale":"2024-03-26T21:50:40.262242+0000","last_undegraded":"2024-03-26T21:50:40.262242+0000","last_fullsized":"2024-03-26T21:50:40.262242+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261159+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:50:40.261159+0000","last_peered":"2024-03-26T21:50:40.261159+0000","last_clean":"2024-03-26T21:50:40.261159+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:50:40.261159+0000","last_undegraded":"2024-03-26T21:50:40.261159+0000","last_fullsized":"2024-03-26T21:50:40.261159+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1052,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262262+0000","last_change":"2024-03-26T21:48:59.014048+0000","last_active":"2024-03-26T21:50:40.262262+0000","last_peered":"2024-03-26T21:50:40.262262+0000","last_clean":"2024-03-26T21:50:40.262262+0000","last_became_active":"2024-03-26T21:48:58.977341+0000","last_became_peered":"2024-03-26T21:48:58.977341+0000","last_unstale":"2024-03-26T21:50:40.262262+0000","last_undegraded":"2024-03-26T21:50:40.262262+0000","last_fullsized":"2024-03-26T21:50:40.262262+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":933,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115103+0000","last_change":"2024-03-26T21:48:40.024866+0000","last_active":"2024-03-26T21:50:25.115103+0000","last_peered":"2024-03-26T21:50:25.115103+0000","last_clean":"2024-03-26T21:50:25.115103+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:50:25.115103+0000","last_undegraded":"2024-03-26T21:50:25.115103+0000","last_fullsized":"2024-03-26T21:50:25.115103+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.024681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.024681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":934,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.283892+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:50:40.283892+0000","last_peered":"2024-03-26T21:50:40.283892+0000","last_clean":"2024-03-26T21:50:40.283892+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:50:40.283892+0000","last_undegraded":"2024-03-26T21:50:40.283892+0000","last_fullsized":"2024-03-26T21:50:40.283892+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":611,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262519+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:50:40.262519+0000","last_peered":"2024-03-26T21:50:40.262519+0000","last_clean":"2024-03-26T21:50:40.262519+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:50:40.262519+0000","last_undegraded":"2024-03-26T21:50:40.262519+0000","last_fullsized":"2024-03-26T21:50:40.262519+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":646,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262439+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:50:40.262439+0000","last_peered":"2024-03-26T21:50:40.262439+0000","last_clean":"2024-03-26T21:50:40.262439+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:50:40.262439+0000","last_undegraded":"2024-03-26T21:50:40.262439+0000","last_fullsized":"2024-03-26T21:50:40.262439+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":948,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272492+0000","last_change":"2024-03-26T21:50:09.758825+0000","last_active":"2024-03-26T21:50:40.272492+0000","last_peered":"2024-03-26T21:50:40.272492+0000","last_clean":"2024-03-26T21:50:40.272492+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:50:40.272492+0000","last_undegraded":"2024-03-26T21:50:40.272492+0000","last_fullsized":"2024-03-26T21:50:40.272492+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:09.758697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:50:09.758697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:13.447825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":580,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.268662+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:50:40.268662+0000","last_peered":"2024-03-26T21:50:40.268662+0000","last_clean":"2024-03-26T21:50:40.268662+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:50:40.268662+0000","last_undegraded":"2024-03-26T21:50:40.268662+0000","last_fullsized":"2024-03-26T21:50:40.268662+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":928,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272380+0000","last_change":"2024-03-26T21:49:33.717279+0000","last_active":"2024-03-26T21:50:40.272380+0000","last_peered":"2024-03-26T21:50:40.272380+0000","last_clean":"2024-03-26T21:50:40.272380+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:50:40.272380+0000","last_undegraded":"2024-03-26T21:50:40.272380+0000","last_fullsized":"2024-03-26T21:50:40.272380+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:33.716995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:49:33.716995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:48.354109+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":678,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261890+0000","last_change":"2024-03-26T21:48:27.673802+0000","last_active":"2024-03-26T21:50:40.261890+0000","last_peered":"2024-03-26T21:50:40.261890+0000","last_clean":"2024-03-26T21:50:40.261890+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:50:40.261890+0000","last_undegraded":"2024-03-26T21:50:40.261890+0000","last_fullsized":"2024-03-26T21:50:40.261890+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:27.673516+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.673516+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":948,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.272685+0000","last_change":"2024-03-26T21:49:48.724032+0000","last_active":"2024-03-26T21:50:40.272685+0000","last_peered":"2024-03-26T21:50:40.272685+0000","last_clean":"2024-03-26T21:50:40.272685+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:50:40.272685+0000","last_undegraded":"2024-03-26T21:50:40.272685+0000","last_fullsized":"2024-03-26T21:50:40.272685+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:48.723773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:49:48.723773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:50:59.774563+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":928,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.271975+0000","last_change":"2024-03-26T21:50:06.681825+0000","last_active":"2024-03-26T21:50:40.271975+0000","last_peered":"2024-03-26T21:50:40.271975+0000","last_clean":"2024-03-26T21:50:40.271975+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:50:40.271975+0000","last_undegraded":"2024-03-26T21:50:40.271975+0000","last_fullsized":"2024-03-26T21:50:40.271975+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_clean_scrub_stamp":"2024-03-26T21:50:06.681696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:09.917933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":901,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.113754+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:50:25.113754+0000","last_peered":"2024-03-26T21:50:25.113754+0000","last_clean":"2024-03-26T21:50:25.113754+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:50:25.113754+0000","last_undegraded":"2024-03-26T21:50:25.113754+0000","last_fullsized":"2024-03-26T21:50:25.113754+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":321,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261464+0000","last_change":"2024-03-26T21:48:59.006965+0000","last_active":"2024-03-26T21:50:40.261464+0000","last_peered":"2024-03-26T21:50:40.261464+0000","last_clean":"2024-03-26T21:50:40.261464+0000","last_became_active":"2024-03-26T21:48:58.977997+0000","last_became_peered":"2024-03-26T21:48:58.977997+0000","last_unstale":"2024-03-26T21:50:40.261464+0000","last_undegraded":"2024-03-26T21:50:40.261464+0000","last_fullsized":"2024-03-26T21:50:40.261464+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:34.910272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.910272+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":271,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.263271+0000","last_change":"2024-03-26T21:48:58.976105+0000","last_active":"2024-03-26T21:50:40.263271+0000","last_peered":"2024-03-26T21:50:40.263271+0000","last_clean":"2024-03-26T21:50:40.263271+0000","last_became_active":"2024-03-26T21:48:58.975714+0000","last_became_peered":"2024-03-26T21:48:58.975714+0000","last_unstale":"2024-03-26T21:50:40.263271+0000","last_undegraded":"2024-03-26T21:50:40.263271+0000","last_fullsized":"2024-03-26T21:50:40.263271+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":610,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261290+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:50:40.261290+0000","last_peered":"2024-03-26T21:50:40.261290+0000","last_clean":"2024-03-26T21:50:40.261290+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:50:40.261290+0000","last_undegraded":"2024-03-26T21:50:40.261290+0000","last_fullsized":"2024-03-26T21:50:40.261290+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":265,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.262083+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:50:40.262083+0000","last_peered":"2024-03-26T21:50:40.262083+0000","last_clean":"2024-03-26T21:50:40.262083+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:50:40.262083+0000","last_undegraded":"2024-03-26T21:50:40.262083+0000","last_fullsized":"2024-03-26T21:50:40.262083+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":654,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261386+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:50:40.261386+0000","last_peered":"2024-03-26T21:50:40.261386+0000","last_clean":"2024-03-26T21:50:40.261386+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:50:40.261386+0000","last_undegraded":"2024-03-26T21:50:40.261386+0000","last_fullsized":"2024-03-26T21:50:40.261386+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":919,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-26T21:50:25.115591+0000","last_change":"2024-03-26T21:48:58.978607+0000","last_active":"2024-03-26T21:50:25.115591+0000","last_peered":"2024-03-26T21:50:25.115591+0000","last_clean":"2024-03-26T21:50:25.115591+0000","last_became_active":"2024-03-26T21:48:58.978253+0000","last_became_peered":"2024-03-26T21:48:58.978253+0000","last_unstale":"2024-03-26T21:50:25.115591+0000","last_undegraded":"2024-03-26T21:50:25.115591+0000","last_fullsized":"2024-03-26T21:50:25.115591+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:33.984501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.984501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":923,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261565+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:50:40.261565+0000","last_peered":"2024-03-26T21:50:40.261565+0000","last_clean":"2024-03-26T21:50:40.261565+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:50:40.261565+0000","last_undegraded":"2024-03-26T21:50:40.261565+0000","last_fullsized":"2024-03-26T21:50:40.261565+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":618,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-26T21:50:40.261718+0000","last_change":"2024-03-26T21:48:44.829403+0000","last_active":"2024-03-26T21:50:40.261718+0000","last_peered":"2024-03-26T21:50:40.261718+0000","last_clean":"2024-03-26T21:50:40.261718+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:50:40.261718+0000","last_undegraded":"2024-03-26T21:50:40.261718+0000","last_fullsized":"2024-03-26T21:50:40.261718+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.829181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.829181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2830303256,"num_objects":43188,"num_object_clones":0,"num_object_copies":129564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4654,"num_objects_misplaced":3251,"num_objects_unfound":0,"num_objects_dirty":43188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194052,"num_write_kb":7591617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30723,"num_bytes_recovered":2013396992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9228398592,"data_stored":9228386376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74580,"ondisk_log_size":74580,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1207,"num_read_kb":1202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":7130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1072,"ondisk_log_size":1072,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973809,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856472,"kb_used_data":1271184,"kb_used_omap":6,"kb_used_meta":585209,"kb_avail":91888680,"statfs":{"total":95995035648,"available":94094008320,"internally_reserved":0,"allocated":1301692416,"data_stored":1301293084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":599254312},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365076,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142292,"kb_used_data":1429448,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91602860,"statfs":{"total":95995035648,"available":93801328640,"internally_reserved":0,"allocated":1463754752,"data_stored":1463361692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575150,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574504,"kb_used_data":1627040,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":91170648,"statfs":{"total":95995035648,"available":93358743552,"internally_reserved":0,"allocated":1666088960,"data_stored":1665672065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072088,"kb_used_data":1156560,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":91673064,"statfs":{"total":95995035648,"available":93873217536,"internally_reserved":0,"allocated":1184317440,"data_stored":1183910830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149986,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2185824,"kb_used_data":1203672,"kb_used_omap":1,"kb_used_meta":982142,"kb_avail":91559328,"statfs":{"total":95995035648,"available":93756751872,"internally_reserved":0,"allocated":1232560128,"data_stored":1232150922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1005713863},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650582,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714848,"kb_used_data":924564,"kb_used_omap":4,"kb_used_meta":790139,"kb_avail":92030304,"statfs":{"total":95995035648,"available":94239031296,"internally_reserved":0,"allocated":946753536,"data_stored":946349128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":809103011},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659415,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1436364,"kb_used_data":711044,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92308788,"statfs":{"total":95995035648,"available":94524198912,"internally_reserved":0,"allocated":728109056,"data_stored":727719850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1160445952,"data_stored":1160445952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1649741824,"data_stored":1649737752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":826081280,"data_stored":826081280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1207173120,"data_stored":1207173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":927731712,"data_stored":927727640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":721879040,"data_stored":721879040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1445728256,"data_stored":1445724184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1289617408,"data_stored":1289617408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:50:47.667 INFO:tasks.thrashosds.thrasher:Setting 5.17 to [1, 2, 3] 2024-03-26T21:50:47.667 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.17', '1', '2', '3'] 2024-03-26T21:50:47.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 5.17 1 2 3 2024-03-26T21:50:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.681 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:47.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:47.783 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:47.823 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:47.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:47.981 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:47.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:48.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:48.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:48.122 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:48.224 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:48.232 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:48.285 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:48.386 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:48.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:48.447 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:48.482 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:48.483 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:50:48.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:48.589 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:48.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:48.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:48.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:48.866 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:48.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:48.967 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.008 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:49.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:49.067 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:49.109 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:49.158 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:49.247 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:49.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.295 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:49.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:49.348 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:50:49.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:49.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[29406]: pgmap v1053: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4863/126075 objects degraded (3.857%); 2891/126075 objects misplaced (2.293%); 3.5 MiB/s, 56 objects/s recovering 2024-03-26T21:50:49.565 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3886590505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:49.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[40587]: pgmap v1053: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4863/126075 objects degraded (3.857%); 2891/126075 objects misplaced (2.293%); 3.5 MiB/s, 56 objects/s recovering 2024-03-26T21:50:49.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3886590505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:49 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:49.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:49 smithi067 ceph-mon[33346]: pgmap v1053: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 4863/126075 objects degraded (3.857%); 2891/126075 objects misplaced (2.293%); 3.5 MiB/s, 56 objects/s recovering 2024-03-26T21:50:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3886590505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:49 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:50:49.839 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:49.877 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:49.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:49.979 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:50.010 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:50.112 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:50.145 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:50.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:50.292 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:50:50.292 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:50:50.354 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:50 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:50:50.355 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:50 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:50.355 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:50 smithi067 ceph-mon[33346]: osdmap e487: 8 total, 8 up, 7 in 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[40587]: osdmap e487: 8 total, 8 up, 7 in 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:50:50.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:50 smithi055 ceph-mon[29406]: osdmap e487: 8 total, 8 up, 7 in 2024-03-26T21:50:50.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:50.424 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:50.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:50.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:50.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:50.702 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:50.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:50.845 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:50.947 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:50.985 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:50.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:51.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:51.135 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:51.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:51.275 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:51.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:51.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:51.518 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:51.565 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:51.666 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[40587]: pgmap v1055: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s; 5059/113346 objects degraded (4.463%); 2555/113346 objects misplaced (2.254%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T21:50:51.667 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:50:51.667 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:50:51.667 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[29406]: pgmap v1055: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s; 5059/113346 objects degraded (4.463%); 2555/113346 objects misplaced (2.254%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T21:50:51.667 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:50:51.667 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:51 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:50:51.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.719 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:51.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:51 smithi067 ceph-mon[33346]: pgmap v1055: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s; 5059/113346 objects degraded (4.463%); 2555/113346 objects misplaced (2.254%); 3.8 MiB/s, 60 objects/s recovering 2024-03-26T21:50:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:51 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:50:51.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:51 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:50:51.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:51.857 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:51.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:51.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:52.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:52.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:52.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:52.399 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:52.477 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:52.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:52.530 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:52.532 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[40587]: osdmap e488: 8 total, 8 up, 7 in 2024-03-26T21:50:52.532 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:50:52.532 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:50:52.532 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5059/113346 objects degraded (4.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:52.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[29406]: osdmap e488: 8 total, 8 up, 7 in 2024-03-26T21:50:52.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:50:52.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:50:52.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:52 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5059/113346 objects degraded (4.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:52.546 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:52.578 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:52.616 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:52.636 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:52.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:52.656 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.678 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:52.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:52.699 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:52.735 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:52 smithi067 ceph-mon[33346]: osdmap e488: 8 total, 8 up, 7 in 2024-03-26T21:50:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:52 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:50:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:52 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:50:52.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:52 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5059/113346 objects degraded (4.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:52.741 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:52.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:52.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:52.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:52.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:53.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:53.108 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:53.141 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T21:50:53.210 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.253 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:53.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:53.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:53.397 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:53.403 INFO:teuthology.orchestra.run.smithi055.stderr:set 5.17 pg_upmap mapping to [1,2,3] 2024-03-26T21:50:53.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:53.533 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:53.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: pgmap v1057: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 5211/111348 objects degraded (4.680%); 2226/111348 objects misplaced (1.999%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1604675427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3677315325' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:50:53.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:53 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:50:53.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:53.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: pgmap v1057: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 5211/111348 objects degraded (4.680%); 2226/111348 objects misplaced (1.999%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1604675427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3677315325' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:50:53.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: pgmap v1057: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 783 op/s; 5211/111348 objects degraded (4.680%); 2226/111348 objects misplaced (1.999%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1604675427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3677315325' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]: dispatch 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:50:53.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:53 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:50:53.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:53.941 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:53.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:54.043 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.081 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:54.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:54.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:54.223 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:54.325 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:54.390 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:54.390 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:54.492 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.521 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:54.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:54.535 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]': finished 2024-03-26T21:50:54.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[40587]: osdmap e489: 8 total, 8 up, 7 in 2024-03-26T21:50:54.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:50:54.536 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]': finished 2024-03-26T21:50:54.536 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[29406]: osdmap e489: 8 total, 8 up, 7 in 2024-03-26T21:50:54.536 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:54 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:50:54.621 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.656 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:54.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:54 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [1, 2, 3]}]': finished 2024-03-26T21:50:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:54 smithi067 ceph-mon[33346]: osdmap e489: 8 total, 8 up, 7 in 2024-03-26T21:50:54.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:54 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:50:54.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:54.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:54.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:55.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:55.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:55.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:55.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:55.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:55.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:55.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:50:55.623 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:55.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:55 smithi067 ceph-mon[33346]: pgmap v1059: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s; 5417/97998 objects degraded (5.528%); 1876/97998 objects misplaced (1.914%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T21:50:55.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:55 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:50:55.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:55.797 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:55 smithi055 ceph-mon[40587]: pgmap v1059: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s; 5417/97998 objects degraded (5.528%); 1876/97998 objects misplaced (1.914%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T21:50:55.797 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:55 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:50:55.797 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:55 smithi055 ceph-mon[29406]: pgmap v1059: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s; 5417/97998 objects degraded (5.528%); 1876/97998 objects misplaced (1.914%); 4.8 MiB/s, 76 objects/s recovering 2024-03-26T21:50:55.797 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:55 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:50:55.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:50:55.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:55.917 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:55.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:56.018 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.049 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:56.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:56.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:50:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:50:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:50:56.312 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:56.354 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:56.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:56.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:56.634 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:56.692 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:56.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:50:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:50:56.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:56.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:57.026 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.073 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:57.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:57.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.233 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:57.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:57.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:57.373 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:57.459 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is set 2024-03-26T21:50:57.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:57.512 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[29406]: pgmap v1060: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 795 B/s rd, 0 B/s wr, 847 op/s; 5417/97008 objects degraded (5.584%); 1876/97008 objects misplaced (1.934%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5417/97008 objects degraded (5.584%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1161544893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[40587]: pgmap v1060: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 795 B/s rd, 0 B/s wr, 847 op/s; 5417/97008 objects degraded (5.584%); 1876/97008 objects misplaced (1.934%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5417/97008 objects degraded (5.584%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1161544893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:57 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:50:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:50:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-mon[33346]: pgmap v1060: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 795 B/s rd, 0 B/s wr, 847 op/s; 5417/97008 objects degraded (5.584%); 1876/97008 objects misplaced (1.934%); 3.1 MiB/s, 49 objects/s recovering 2024-03-26T21:50:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:50:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5417/97008 objects degraded (5.584%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:50:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1161544893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T21:50:57.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:57.807 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:57.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:57.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:50:58.055 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:58.092 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:58.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:50:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:50:58.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:58.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:58.321 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-26T21:50:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-26T21:50:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-26T21:50:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-26T21:50:58.321 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-26T21:50:58.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:58.367 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:58.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.512 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:58.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:58.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: osdmap e490: 8 total, 8 up, 7 in 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: osdmap e490: 8 total, 8 up, 7 in 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:58.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:58.641 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:50:58.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T21:50:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: osdmap e490: 8 total, 8 up, 7 in 2024-03-26T21:50:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:50:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:50:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:58 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:50:58.743 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:58.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:58.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:58.934 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:58.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:59.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:59.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:59.171 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:50:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.209 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:59.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:50:59.264 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:50:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:50:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:50:59.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:50:59.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:50:59.291 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-26T21:50:59.291 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T21:50:59.291 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T21:50:59.292 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-26T21:50:59.310 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:50:59.426 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:50:59.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:59.521 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:50:59 smithi055 ceph-mon[40587]: pgmap v1062: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 796 B/s rd, 0 B/s wr, 1.01k op/s; 5601/93810 objects degraded (5.971%); 1511/93810 objects misplaced (1.611%); 4.9 MiB/s, 78 objects/s recovering 2024-03-26T21:50:59.522 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:59.522 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:50:59 smithi055 ceph-mon[29406]: pgmap v1062: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 796 B/s rd, 0 B/s wr, 1.01k op/s; 5601/93810 objects degraded (5.971%); 1511/93810 objects misplaced (1.611%); 4.9 MiB/s, 78 objects/s recovering 2024-03-26T21:50:59.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:59.560 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:59.660 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:50:59.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:50:59.692 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:50:59.705 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:50:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:50:59.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:50:59 smithi067 ceph-mon[33346]: pgmap v1062: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 796 B/s rd, 0 B/s wr, 1.01k op/s; 5601/93810 objects degraded (5.971%); 1511/93810 objects misplaced (1.611%); 4.9 MiB/s, 78 objects/s recovering 2024-03-26T21:50:59.745 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:50:59.774 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:50:59.790 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:50:59.793 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:50:59.833 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:50:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.856 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:50:59.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:50:59.874 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:50:59.913 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:50:59.952 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:50:59.955 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:50:59.955 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:50:59.955 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:50:59.955 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:50:59.955 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:50:59.956 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:50:59.956 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:50:59.956 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:50:59.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:50:59.985 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:50:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:50:59.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:00.087 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:00.112 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:00.214 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:00.238 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:00.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T21:51:00.340 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:00.387 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:00.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:00.596 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4284583750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:00.597 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4284583750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:00.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:00.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4284583750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:00.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:00.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:00.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:00.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:01.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.070 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:01.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:01.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:01.206 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:01.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:01.342 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:01.446 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:01.482 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:01.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:51:01.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:01.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:01.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:01 smithi067 ceph-mon[33346]: pgmap v1063: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5801/80754 objects degraded (7.184%); 1161/80754 objects misplaced (1.438%); 3.7 MiB/s, 59 objects/s recovering 2024-03-26T21:51:01.769 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:01.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:01 smithi055 ceph-mon[40587]: pgmap v1063: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5801/80754 objects degraded (7.184%); 1161/80754 objects misplaced (1.438%); 3.7 MiB/s, 59 objects/s recovering 2024-03-26T21:51:01.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:01 smithi055 ceph-mon[29406]: pgmap v1063: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 5801/80754 objects degraded (7.184%); 1161/80754 objects misplaced (1.438%); 3.7 MiB/s, 59 objects/s recovering 2024-03-26T21:51:01.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:01.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:01.940 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:51:02.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:02.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:02.183 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:02.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:02.324 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:02.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:02.455 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:02.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:02.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:02.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:02.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:02.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:02 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5801/80754 objects degraded (7.184%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:02.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:02 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5801/80754 objects degraded (7.184%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:02.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:02 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5801/80754 objects degraded (7.184%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:02.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:02.874 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:02.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:02.976 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:03.116 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:03.147 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:03.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.287 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:03.387 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.423 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:03.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:03.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:03.696 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:03 smithi067 ceph-mon[33346]: pgmap v1064: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 5922/78816 objects degraded (7.514%); 849/78816 objects misplaced (1.077%); 5.0 MiB/s, 79 objects/s recovering 2024-03-26T21:51:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:03.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:03.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:03 smithi055 ceph-mon[40587]: pgmap v1064: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 5922/78816 objects degraded (7.514%); 849/78816 objects misplaced (1.077%); 5.0 MiB/s, 79 objects/s recovering 2024-03-26T21:51:03.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:03 smithi055 ceph-mon[29406]: pgmap v1064: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 5922/78816 objects degraded (7.514%); 849/78816 objects misplaced (1.077%); 5.0 MiB/s, 79 objects/s recovering 2024-03-26T21:51:03.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:03.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:03.940 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:03.969 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:03.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:04.012 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:51:04.012 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:51:04.071 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.103 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:04.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:04.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:04.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:04.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:04.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:04.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.523 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:04.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:04.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:04 smithi055 ceph-mon[40587]: pgmap v1065: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6091/66042 objects degraded (9.223%); 529/66042 objects misplaced (0.801%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T21:51:04.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:04 smithi055 ceph-mon[29406]: pgmap v1065: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6091/66042 objects degraded (9.223%); 529/66042 objects misplaced (0.801%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T21:51:04.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:04.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:04.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:04 smithi067 ceph-mon[33346]: pgmap v1065: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6091/66042 objects degraded (9.223%); 529/66042 objects misplaced (0.801%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T21:51:04.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:04.796 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:04.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:04.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:05.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T21:51:05.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:05.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:05.213 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:05.315 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:05.355 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:05.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:05.493 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:05.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:05.632 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:05.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:05.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:05 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:05.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:05 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:05.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:05 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:05.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:05 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:05.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:05.908 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:05.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:05 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:05.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:05 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:06.011 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:06.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:06.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:06.188 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:06.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:06.430 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:06.457 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:06.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:06.596 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:06.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:06.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:06 smithi055 ceph-mon[40587]: osdmap e491: 8 total, 8 up, 7 in 2024-03-26T21:51:06.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:06 smithi055 ceph-mon[40587]: pgmap v1067: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 6091/65094 objects degraded (9.357%); 529/65094 objects misplaced (0.813%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:51:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:06 smithi055 ceph-mon[29406]: osdmap e491: 8 total, 8 up, 7 in 2024-03-26T21:51:06.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:06 smithi055 ceph-mon[29406]: pgmap v1067: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 6091/65094 objects degraded (9.357%); 529/65094 objects misplaced (0.813%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:51:06.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:06.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:06.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:06 smithi067 ceph-mon[33346]: osdmap e491: 8 total, 8 up, 7 in 2024-03-26T21:51:06.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:06 smithi067 ceph-mon[33346]: pgmap v1067: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 6091/65094 objects degraded (9.357%); 529/65094 objects misplaced (0.813%); 3.7 MiB/s, 58 objects/s recovering 2024-03-26T21:51:07.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:07.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:07.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:07.270 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:07.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:07.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:07.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:07.647 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.684 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:07.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:07.784 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:07 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6091/65094 objects degraded (9.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:07.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:07.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:07 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6091/65094 objects degraded (9.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:07.816 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:07 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6091/65094 objects degraded (9.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.822 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:07.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:07.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:07.959 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:07.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:08.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.090 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:08.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:08.192 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:08.231 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:08.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:08.332 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:08.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:08.464 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.502 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:08.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:08.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:08.639 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:08.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:08.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:08 smithi055 ceph-mon[40587]: pgmap v1068: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6269/61890 objects degraded (10.129%); 165/61890 objects misplaced (0.267%); 4.1 MiB/s, 67 objects/s recovering 2024-03-26T21:51:08.815 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:08.815 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:08 smithi055 ceph-mon[29406]: pgmap v1068: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6269/61890 objects degraded (10.129%); 165/61890 objects misplaced (0.267%); 4.1 MiB/s, 67 objects/s recovering 2024-03-26T21:51:08.816 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:08.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:08.918 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:08.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:08 smithi067 ceph-mon[33346]: pgmap v1068: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 6269/61890 objects degraded (10.129%); 165/61890 objects misplaced (0.267%); 4.1 MiB/s, 67 objects/s recovering 2024-03-26T21:51:08.989 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:09.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.042 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:09.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:09.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:09.176 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:09.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:09.315 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:09.316 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:09.418 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:09.459 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:09.560 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:09.597 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:09.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:09.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:09.840 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:09.878 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:09.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:09.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:10.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:10.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:10.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:10.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:10.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:10.661 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:10.704 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:10.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:10.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:10.979 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:10.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:10.992 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[40587]: pgmap v1069: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1000 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 6267/48585 objects degraded (12.899%); 3.8 MiB/s, 69 objects/s recovering 2024-03-26T21:51:10.992 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:10.992 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:10.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[29406]: pgmap v1069: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1000 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 6267/48585 objects degraded (12.899%); 3.8 MiB/s, 69 objects/s recovering 2024-03-26T21:51:10.992 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:10.993 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:10 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:11.035 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:11.067 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:11.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:11.136 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:11.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:10 smithi067 ceph-mon[33346]: pgmap v1069: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 1000 MiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 6267/48585 objects degraded (12.899%); 3.8 MiB/s, 69 objects/s recovering 2024-03-26T21:51:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:10 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:11.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:10 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:11.280 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:11.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:11.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:11.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:11.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:11.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:11.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:11.799 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:11.843 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:11.945 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:11.946 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:51:11.947 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "" 2024-03-26T21:51:11.947 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:51:11.983 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:11.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:12.041 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:11 smithi055 ceph-mon[40587]: osdmap e492: 8 total, 8 up, 7 in 2024-03-26T21:51:12.043 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:11 smithi055 ceph-mon[29406]: osdmap e492: 8 total, 8 up, 7 in 2024-03-26T21:51:12.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:12.121 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:12.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:12.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:11 smithi067 ceph-mon[33346]: osdmap e492: 8 total, 8 up, 7 in 2024-03-26T21:51:12.253 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:12.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:12.394 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:12.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:12.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:12.667 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:12.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:12.803 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:12.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:12.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:13.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:13.057 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T21:51:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:13.181 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:13.185 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:12 smithi055 ceph-mon[40587]: pgmap v1071: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 957 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 6027/46491 objects degraded (12.964%); 2.0 MiB/s, 74 objects/s recovering 2024-03-26T21:51:13.185 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:12 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 6027/46491 objects degraded (12.964%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:13.187 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:12 smithi055 ceph-mon[29406]: pgmap v1071: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 957 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 6027/46491 objects degraded (12.964%); 2.0 MiB/s, 74 objects/s recovering 2024-03-26T21:51:13.187 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:12 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 6027/46491 objects degraded (12.964%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:12 smithi067 ceph-mon[33346]: pgmap v1071: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 957 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 6027/46491 objects degraded (12.964%); 2.0 MiB/s, 74 objects/s recovering 2024-03-26T21:51:13.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:12 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 6027/46491 objects degraded (12.964%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:13.248 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:13.349 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:13.394 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:13.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:13.542 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:13.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:13.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:13.821 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:13.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:13.959 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:13.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:14.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:14.098 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:14.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:14.244 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:14.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:14.385 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:14.419 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:51:14.420 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:51:14.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:14.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:14.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:14.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:14.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.788 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:14.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:14.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:14.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:14.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:15.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:15.066 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:15.168 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.204 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:15.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:15.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:14 smithi067 ceph-mon[33346]: pgmap v1072: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 660 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 5850/32265 objects degraded (18.131%); 2.0 MiB/s, 93 objects/s recovering 2024-03-26T21:51:15.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:15.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:14 smithi055 ceph-mon[40587]: pgmap v1072: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 660 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 5850/32265 objects degraded (18.131%); 2.0 MiB/s, 93 objects/s recovering 2024-03-26T21:51:15.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:14 smithi055 ceph-mon[29406]: pgmap v1072: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 660 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 5850/32265 objects degraded (18.131%); 2.0 MiB/s, 93 objects/s recovering 2024-03-26T21:51:15.343 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:15.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.483 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:15.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:15.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:15.614 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:15.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:15.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:15.863 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:15.896 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:15.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:15.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:16.031 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:16.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:16.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:16.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:16.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:16.580 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:16.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:16.726 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:16.828 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:16.870 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:16.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:16.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:17.011 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:17.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:17.154 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:17.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:16 smithi067 ceph-mon[33346]: pgmap v1073: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 638 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 5850/31209 objects degraded (18.745%); 1.6 MiB/s, 76 objects/s recovering 2024-03-26T21:51:17.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:17.291 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:17.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:16 smithi055 ceph-mon[40587]: pgmap v1073: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 638 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 5850/31209 objects degraded (18.745%); 1.6 MiB/s, 76 objects/s recovering 2024-03-26T21:51:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:16 smithi055 ceph-mon[29406]: pgmap v1073: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 638 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 5850/31209 objects degraded (18.745%); 1.6 MiB/s, 76 objects/s recovering 2024-03-26T21:51:17.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:17.422 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:17.524 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:17.577 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:17.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.714 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:17.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:17.815 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:17.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:17.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:18.004 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:18.048 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:17 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5850/31209 objects degraded (18.745%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:18.049 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:17 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5850/31209 objects degraded (18.745%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:18.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:18.113 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T21:51:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:18.239 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:18.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:17 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5850/31209 objects degraded (18.745%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:18.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:18.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.295 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:18.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:18.396 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.468 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:18.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:18.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.617 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:18.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:18.681 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:18.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:18.748 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 6, 7, 4, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-26T21:51:18.748 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T21:51:18.748 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T21:51:18.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T21:51:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.754 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:18.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:18.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:18.924 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:18.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[40587]: pgmap v1074: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 575 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 5595/28182 objects degraded (19.853%); 557 KiB/s, 83 objects/s recovering 2024-03-26T21:51:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:18.988 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[29406]: pgmap v1074: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 575 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 5595/28182 objects degraded (19.853%); 557 KiB/s, 83 objects/s recovering 2024-03-26T21:51:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:18.989 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:19.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:19.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:19.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.212 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:19.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:18 smithi067 ceph-mon[33346]: pgmap v1074: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 575 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 5595/28182 objects degraded (19.853%); 557 KiB/s, 83 objects/s recovering 2024-03-26T21:51:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:18 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:19.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:18 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:19.268 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:19.313 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:19.349 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:19.450 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:19.487 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.489 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:19.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:19.590 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:19.634 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:19.736 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.776 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:19.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:19.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:19.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:19.938 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is set 2024-03-26T21:51:20.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:20.049 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:20.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.194 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:20.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:19 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/633763897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:19 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:20.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:19 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/633763897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:19 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:19 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/633763897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:19 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:51:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.332 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:20.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:20.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:20.479 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:20.581 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:20.609 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:20.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:20.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:20.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:20.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:20.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:21.025 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:21.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:21.158 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: pgmap v1075: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 285 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5413/14247 objects degraded (37.994%); 0 B/s, 85 objects/s recovering 2024-03-26T21:51:21.158 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:21.158 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[40587]: osdmap e493: 8 total, 8 up, 7 in 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: pgmap v1075: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 285 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5413/14247 objects degraded (37.994%); 0 B/s, 85 objects/s recovering 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:51:21.159 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:20 smithi055 ceph-mon[29406]: osdmap e493: 8 total, 8 up, 7 in 2024-03-26T21:51:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.162 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:21.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: pgmap v1075: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 285 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5413/14247 objects degraded (37.994%); 0 B/s, 85 objects/s recovering 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T21:51:21.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:20 smithi067 ceph-mon[33346]: osdmap e493: 8 total, 8 up, 7 in 2024-03-26T21:51:21.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:21.305 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:21.407 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:21.446 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:21.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:21.588 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:21.689 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:21.723 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:21.824 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:21.868 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:21.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:21.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:22.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.147 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:22.231 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Removed 118619 objects 2024-03-26T21:51:22.231 INFO:tasks.radosbench.radosbench.0.smithi055.stdout:Clean up completed and total clean up time :120.279 2024-03-26T21:51:22.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:22.272 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:22.374 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:22.399 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:22.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:22.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:22.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:22.878 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:22.903 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:22.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:23.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:23.029 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:23.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:23.155 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:22 smithi067 ceph-mon[33346]: pgmap v1077: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 241 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5160/12117 objects degraded (42.585%); 0 B/s, 86 objects/s recovering 2024-03-26T21:51:23.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:22 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 5160/12117 objects degraded (42.585%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:23.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:23.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:22 smithi055 ceph-mon[29406]: pgmap v1077: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 241 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5160/12117 objects degraded (42.585%); 0 B/s, 86 objects/s recovering 2024-03-26T21:51:23.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:22 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 5160/12117 objects degraded (42.585%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:23.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:22 smithi055 ceph-mon[40587]: pgmap v1077: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 241 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s; 5160/12117 objects degraded (42.585%); 0 B/s, 86 objects/s recovering 2024-03-26T21:51:23.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:22 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 5160/12117 objects degraded (42.585%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:23.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:23.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:23.635 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:23.659 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:23.761 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:23.787 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:23.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:23.914 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:23.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:24.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:24.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:24.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:24.164 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:24.266 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:24.291 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:24.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:24.416 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:24.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:24.543 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:24.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:24.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:24.795 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:24.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:24.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:24.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:25.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:25.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:25.173 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:25.229 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-26T21:51:25.229 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-26T21:51:25.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:24 smithi067 ceph-mon[33346]: pgmap v1078: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 22 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 4760/1602 objects degraded (297.129%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:25.254 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:24 smithi055 ceph-mon[40587]: pgmap v1078: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 22 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 4760/1602 objects degraded (297.129%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:25.254 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:24 smithi055 ceph-mon[29406]: pgmap v1078: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 22 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 4760/1602 objects degraded (297.129%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:25.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:25.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:25.427 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:25.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:25.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:25.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:25.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:25.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:26.033 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:26.058 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:26.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:26.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:26.184 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:26.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:26.312 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:26.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:26.438 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:26.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:26.565 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:26.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:26.691 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:26.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:26.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:26.944 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:26.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:27.020 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:26 smithi055 ceph-mon[40587]: pgmap v1079: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 6.0 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 4760/858 objects degraded (554.779%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:27.021 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:26 smithi055 ceph-mon[29406]: pgmap v1079: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 6.0 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 4760/858 objects degraded (554.779%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:27.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:27.070 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:27.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:27.197 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:27.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:26 smithi067 ceph-mon[33346]: pgmap v1079: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 6.0 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 4760/858 objects degraded (554.779%); 0 B/s, 108 objects/s recovering 2024-03-26T21:51:27.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:27.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:27.452 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:27.554 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:27.560 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:27.581 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:27.683 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:27.710 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:27.812 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:27.840 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:27.844 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:51:27.874 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:51:27.897 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:51:27.924 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:51:27.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:27.949 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:51:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:27.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:27.974 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:51:27.994 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:51:28.018 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:51:28.020 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:51:28.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:27 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 4760/858 objects degraded (554.779%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:28.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3742413948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:27 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 4760/858 objects degraded (554.779%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:28.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3742413948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:28.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:28.195 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:28.220 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:27 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 4760/858 objects degraded (554.779%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:28.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3742413948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:28.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:28.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:28.346 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:28.448 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:28.473 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:28.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:28.599 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:28.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:28.725 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:28.756 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:51:28.801 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:51:28.801 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [], 2024-03-26T21:51:28.801 INFO:teuthology.orchestra.run.smithi055.stdout: "complaint_time": 30, 2024-03-26T21:51:28.801 INFO:teuthology.orchestra.run.smithi055.stdout: "num_blocked_ops": 0 2024-03-26T21:51:28.801 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:51:28.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:28.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:28.981 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:28.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:29.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:29.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:28 smithi067 ceph-mon[33346]: pgmap v1080: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 3717/594 objects degraded (625.758%); 0 B/s, 187 objects/s recovering 2024-03-26T21:51:29.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:28 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:28 smithi055 ceph-mon[29406]: pgmap v1080: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 3717/594 objects degraded (625.758%); 0 B/s, 187 objects/s recovering 2024-03-26T21:51:29.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:28 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:29.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:28 smithi055 ceph-mon[40587]: pgmap v1080: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 3717/594 objects degraded (625.758%); 0 B/s, 187 objects/s recovering 2024-03-26T21:51:29.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:28 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:29.316 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:29.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:29.359 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:29.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:29.485 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:29.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:29.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:29.738 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:29.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:29.966 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:29.994 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:29.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:30.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:30.112 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.120 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:30.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:30.144 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T21:51:30.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:30.244 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:30.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.370 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:30.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:30.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:30.496 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:30.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:30.622 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:30.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:30.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:30.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:30.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:31.102 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.130 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:31.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:31.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:30 smithi067 ceph-mon[33346]: pgmap v1081: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 455 op/s; 2682/594 objects degraded (451.515%); 0 B/s, 273 objects/s recovering 2024-03-26T21:51:31.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:30 smithi067 ceph-mon[33346]: osdmap e494: 8 total, 8 up, 7 in 2024-03-26T21:51:31.255 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:31.276 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[29406]: pgmap v1081: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 455 op/s; 2682/594 objects degraded (451.515%); 0 B/s, 273 objects/s recovering 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[29406]: osdmap e494: 8 total, 8 up, 7 in 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[40587]: pgmap v1081: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 455 op/s; 2682/594 objects degraded (451.515%); 0 B/s, 273 objects/s recovering 2024-03-26T21:51:31.277 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:30 smithi055 ceph-mon[40587]: osdmap e494: 8 total, 8 up, 7 in 2024-03-26T21:51:31.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:31.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:31.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:31.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.633 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:31.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:31.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:31.759 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:31.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:31.885 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:31.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:31.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:32.015 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:32.117 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:32.141 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:31 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:31 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:31 smithi067 ceph-mon[33346]: osdmap e495: 8 total, 8 up, 7 in 2024-03-26T21:51:32.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:31 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 2682/594 objects degraded (451.515%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:32.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:32.265 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:32.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[40587]: osdmap e495: 8 total, 8 up, 7 in 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 2682/594 objects degraded (451.515%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[29406]: osdmap e495: 8 total, 8 up, 7 in 2024-03-26T21:51:32.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:31 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 2682/594 objects degraded (451.515%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T21:51:32.368 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.392 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:32.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:32.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:32.517 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:32.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:32.644 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:32.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:32.769 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:32.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:32.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:32.996 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.020 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:33.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:33.121 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:33.145 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:33.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:32 smithi067 ceph-mon[33346]: pgmap v1084: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 42 op/s; 1590/594 objects degraded (267.677%); 0 B/s, 396 objects/s recovering 2024-03-26T21:51:33.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:33.271 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:33.315 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:32 smithi055 ceph-mon[40587]: pgmap v1084: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 42 op/s; 1590/594 objects degraded (267.677%); 0 B/s, 396 objects/s recovering 2024-03-26T21:51:33.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:32 smithi055 ceph-mon[29406]: pgmap v1084: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 42 op/s; 1590/594 objects degraded (267.677%); 0 B/s, 396 objects/s recovering 2024-03-26T21:51:33.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:33.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T21:51:33.397 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:33.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:33.522 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:33.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.649 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:33.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:33.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:33.774 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:33.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:33.900 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:33.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:34.001 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:34.029 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:34.131 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:34.156 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:34.257 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.283 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:34.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:34.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:51:34.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:51:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:51:34.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:34 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:51:34.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:34 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:34.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:34 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:51:34.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:34 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:51:34.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:34.540 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:34.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:34.771 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:34.799 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:34.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:34.929 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:34.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:35.031 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:35.057 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:35.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:35.187 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:35.276 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:35 smithi055 ceph-mon[40587]: pgmap v1085: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 594 objects/s recovering 2024-03-26T21:51:35.276 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:35 smithi055 ceph-mon[40587]: osdmap e496: 8 total, 8 up, 7 in 2024-03-26T21:51:35.276 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:35 smithi055 ceph-mon[29406]: pgmap v1085: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 594 objects/s recovering 2024-03-26T21:51:35.277 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:35 smithi055 ceph-mon[29406]: osdmap e496: 8 total, 8 up, 7 in 2024-03-26T21:51:35.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:35.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:35.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:35.446 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:35.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:35 smithi067 ceph-mon[33346]: pgmap v1085: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 594 objects/s recovering 2024-03-26T21:51:35.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:35 smithi067 ceph-mon[33346]: osdmap e496: 8 total, 8 up, 7 in 2024-03-26T21:51:35.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:35.574 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:35.630 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:35.630 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:35.631 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:35.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:35.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:35.841 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:51:35.841 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:51:35.841 INFO:teuthology.orchestra.run.smithi055.stdout: "epoch": 497, 2024-03-26T21:51:35.841 INFO:teuthology.orchestra.run.smithi055.stdout: "fsid": "e5f52b02-ebb6-11ee-95d0-87774f69a715", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "created": "2024-03-26T21:22:56.673257+0000", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "modified": "2024-03-26T21:51:35.004748+0000", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "last_up_change": "2024-03-26T21:41:24.572964+0000", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "last_in_change": "2024-03-26T21:48:57.950733+0000", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_num": 5806080, 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_set": [ 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "nodeep-scrub", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "noscrub", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "pglog_hardlimit", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "purged_snapdirs", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_deletes", 2024-03-26T21:51:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: "sortbitwise" 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_version": 20, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_snapshot": "", 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_max": 11, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "max_osd": 8, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "require_min_compat_client": "luminous", 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "min_compat_client": "luminous", 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "require_osd_release": "squid", 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "allow_crimson": false, 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: "pools": [ 2024-03-26T21:51:35.843 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 1, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".mgr", 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:26:13.884991+0000", 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.844 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 1, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 1, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 1, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 1, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 1, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.845 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "484", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.846 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:51:35.848 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_max": 32, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_min": 1 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": {} 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 6.9800000190734863, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 6.9800000190734863, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.43000000715255737, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 3, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 3, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.849 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 2, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "datapool", 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.142617+0000", 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 8193, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.850 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 3, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 3, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 3, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 3, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 3, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.851 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "71", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 2, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 71, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.852 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.853 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.854 INFO:teuthology.orchestra.run.smithi055.stdout: "rbd": {} 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.3299999237060547, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.3299999237060547, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 0.86000001430511475, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.855 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 3, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": ".rgw.root", 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:33.591599+0000", 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.856 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.857 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "68", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.858 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.859 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:51:35.860 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.5299999713897705, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.5299999713897705, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 4, 2024-03-26T21:51:35.861 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.log", 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:34.904126+0000", 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.862 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "70", 2024-03-26T21:51:35.863 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.864 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.865 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.866 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.190000057220459, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.190000057220459, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 5, 2024-03-26T21:51:35.867 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.control", 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:36.852373+0000", 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.868 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "72", 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.869 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.870 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.871 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.5299999713897705, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.5299999713897705, 2024-03-26T21:51:35.872 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:51:35.873 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T21:51:35.873 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T21:51:35.873 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.873 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 6, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "default.rgw.meta", 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:28:38.937273+0000", 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.874 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 32, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 32, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 32, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 32, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 32, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.875 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "74", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.876 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.877 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.878 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "options": { 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_bias": 4 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "rgw": {} 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 1.75, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 1.75, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 1.75, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 1.75, 2024-03-26T21:51:35.879 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "pool": 11, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_name": "unique_pool_4", 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "create_time": "2024-03-26T21:47:37.048063+0000", 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "flags": 1, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "flags_names": "hashpspool", 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "type": 1, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 3, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "min_size": 2, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_rule": 0, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_count": 0, 2024-03-26T21:51:35.880 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_target": 0, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "object_hash": 2, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_autoscale_mode": "off", 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num": 16, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num": 16, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_placement_num_target": 16, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_target": 16, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_num_pending": 16, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "last_pg_merge_meta": { 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "source_pgid": "0.0", 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "ready_epoch": 0, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_started": 0, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "last_epoch_clean": 0, 2024-03-26T21:51:35.881 INFO:teuthology.orchestra.run.smithi055.stdout: "source_version": "0'0", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "target_version": "0'0" 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "last_change": "439", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend": "0", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "auid": 0, 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_mode": "selfmanaged", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_seq": 0, 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "snap_epoch": 0, 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "pool_snaps": [], 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps": "[]", 2024-03-26T21:51:35.882 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_bytes": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "quota_max_objects": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "tiers": [], 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "tier_of": -1, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "read_tier": -1, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "write_tier": -1, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_mode": "none", 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_bytes": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "target_max_objects": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_flush_age": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "cache_min_evict_age": 0, 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profile": "", 2024-03-26T21:51:35.883 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_params": { 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "none" 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_period": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_count": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "use_gmt_hitset": true, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "min_read_recency_for_promote": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "min_write_recency_for_promote": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "hit_set_search_last_n": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "grade_table": [], 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "stripe_width": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "expected_num_objects": 0, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "fast_read": false, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "options": {}, 2024-03-26T21:51:35.884 INFO:teuthology.orchestra.run.smithi055.stdout: "application_metadata": { 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "rados": {} 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "read_balance": { 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "score_type": "Fair distribution", 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "score_acting": 2.630000114440918, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "score_stable": 2.630000114440918, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "optimal_score": 1, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_acting": 2.630000114440918, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "raw_score_stable": 2.630000114440918, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity_weighted": 0.875, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity": 1, 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.885 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4f487071-c151-406b-b25b-3135cb3c63ae", 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 8, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 471, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.886 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6801", 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6802", 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:51:35.887 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6804", 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.888 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6803", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 317381212 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6801/317381212", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6802/317381212", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6804/317381212", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6803/317381212", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "4fbc90fe-926a-4ac6-b5f0-d483ba4c09db", 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 13, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 481, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.890 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6805", 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6806", 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:51:35.891 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6808", 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.892 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6807", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3733554168 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6805/3733554168", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6806/3733554168", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6808/3733554168", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6807/3733554168", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "e2fe8018-1060-499c-a7e6-56d4da0eba7b", 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 0, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 0, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 18, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 433, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.894 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6809", 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6810", 2024-03-26T21:51:35.895 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6812", 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.896 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6811", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3059242923 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6809/3059242923", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6810/3059242923", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6812/3059242923", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6811/3059242923", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "30637a8d-5c1b-4ede-8214-a4595c875d1b", 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 26, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 496, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.898 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6813", 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.899 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6814", 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6816", 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.900 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.55:6815", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3851321468 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.55:6813/3851321468", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.55:6814/3851321468", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.55:6816/3851321468", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.55:6815/3851321468", 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.901 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "cd2d68be-4b10-4242-826b-e085c2ce5598", 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 31, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 92, 2024-03-26T21:51:35.902 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 133, 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 494, 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 93, 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6800", 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.903 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6801", 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6803", 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:51:35.904 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6802", 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 254752278 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6800/254752278", 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6801/254752278", 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6803/254752278", 2024-03-26T21:51:35.905 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6802/254752278", 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "7468b94a-3216-4bc2-9b57-1d93e7c0b213", 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.906 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 41, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 467, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6804", 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.907 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6805", 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6807", 2024-03-26T21:51:35.908 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6806", 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 278711358 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6804/278711358", 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6805/278711358", 2024-03-26T21:51:35.909 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6807/278711358", 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6806/278711358", 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "d47c79f3-96bf-46b6-a7f3-a315446d04b9", 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 0, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 0, 2024-03-26T21:51:35.910 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 0, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 50, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 433, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 0, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6808", 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.911 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6809", 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6811", 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:51:35.912 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6810", 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 3926251639 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6808/3926251639", 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6809/3926251639", 2024-03-26T21:51:35.913 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6811/3926251639", 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6810/3926251639", 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "uuid": "f10b98b8-887d-4f83-8bc9-99f5c302dc55", 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "up": 1, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "in": 1, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "weight": 1, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_affinity": 1, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_begin": 59, 2024-03-26T21:51:35.914 INFO:teuthology.orchestra.run.smithi055.stdout: "last_clean_end": 248, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "up_from": 276, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "up_thru": 476, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "down_at": 249, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "lost_at": 0, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addrs": { 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6812", 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.915 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addrs": { 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6813", 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addrs": { 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6815", 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.916 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addrs": { 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "addrvec": [ 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "type": "v2", 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "addr": "172.21.15.67:6814", 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "nonce": 650373445 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "public_addr": "172.21.15.67:6812/650373445", 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "cluster_addr": "172.21.15.67:6813/650373445", 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_back_addr": "172.21.15.67:6815/650373445", 2024-03-26T21:51:35.917 INFO:teuthology.orchestra.run.smithi055.stdout: "heartbeat_front_addr": "172.21.15.67:6814/650373445", 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "state": [ 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "exists", 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "up" 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_xinfo": [ 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 0, 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.918 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:33.212103+0000", 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 1, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:05.267779+0000", 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.919 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 2, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 65536, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:19.896579+0000", 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 3, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.920 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:09.854116+0000", 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 4, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:29.231740+0000", 2024-03-26T21:51:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 92 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 5, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:51:30.396887+0000", 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 6, 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.922 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:50:50.350586+0000", 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 0 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": 7, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "down_stamp": "0.000000", 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_probability": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "laggy_interval": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "features": 4540701547738038271, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "old_weight": 0, 2024-03-26T21:51:35.923 INFO:teuthology.orchestra.run.smithi055.stdout: "last_purged_snaps_scrub": "2024-03-26T21:50:53.113466+0000", 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: "dead_epoch": 248 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap": [ 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "5.17", 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: "osds": [ 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: 1, 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: 2, 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: 3 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_items": [ 2024-03-26T21:51:35.924 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.4", 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.6", 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 5 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.925 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "pgid": "4.18", 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "mappings": [ 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "from": 1, 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "to": 0 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: ], 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_upmap_primaries": [], 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "pg_temp": [], 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "primary_temp": [], 2024-03-26T21:51:35.926 INFO:teuthology.orchestra.run.smithi055.stdout: "blocklist": { 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2649740776": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/4141219490": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/147833426": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2854772543": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/2804789175": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2199694469": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1614234257": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2834138692": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/1740219825": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/268664696": "2024-03-27T21:23:57.787077+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:6800/536717527": "2024-03-27T21:23:30.740110+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/966571165": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/2757111431": "2024-03-27T21:29:15.251747+0000", 2024-03-26T21:51:35.927 INFO:teuthology.orchestra.run.smithi055.stdout: "172.21.15.55:0/559658018": "2024-03-27T21:23:57.787077+0000" 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "range_blocklist": {}, 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "erasure_code_profiles": { 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "default": { 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "crush-failure-domain": "osd", 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "k": "2", 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "m": "1", 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "plugin": "jerasure", 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "technique": "reed_sol_van" 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "removed_snaps_queue": [], 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "new_removed_snaps": [], 2024-03-26T21:51:35.928 INFO:teuthology.orchestra.run.smithi055.stdout: "new_purged_snaps": [], 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "crush_node_flags": {}, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "device_class_flags": {}, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode": { 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_enabled": false, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_bucket_count": 0, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "degraded_stretch_mode": 0, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "recovering_stretch_mode": 0, 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: "stretch_mode_bucket": 0 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:35.929 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:51:35.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:35.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:36.062 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:36.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:36 smithi055 ceph-mon[40587]: osdmap e497: 8 total, 8 up, 7 in 2024-03-26T21:51:36.167 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:36 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1450180267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:51:36.167 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:36 smithi055 ceph-mon[29406]: osdmap e497: 8 total, 8 up, 7 in 2024-03-26T21:51:36.167 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:36 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1450180267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:51:36.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:36.219 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:36.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:36.348 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:36.450 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.475 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:36.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:36 smithi067 ceph-mon[33346]: osdmap e497: 8 total, 8 up, 7 in 2024-03-26T21:51:36.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:36 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1450180267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T21:51:36.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:36.599 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:36.701 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:36.725 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:36.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:36.855 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:36.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:36.982 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:36.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:37.084 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:37.107 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:37.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:37.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:37 smithi055 ceph-mon[40587]: pgmap v1088: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 880 B/s rd, 0 op/s; 0 B/s, 460 objects/s recovering 2024-03-26T21:51:37.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:37 smithi055 ceph-mon[40587]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T21:51:37.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:37 smithi055 ceph-mon[29406]: pgmap v1088: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 880 B/s rd, 0 op/s; 0 B/s, 460 objects/s recovering 2024-03-26T21:51:37.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:37 smithi055 ceph-mon[29406]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T21:51:37.338 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.362 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:37.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:37.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:37 smithi067 ceph-mon[33346]: pgmap v1088: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 880 B/s rd, 0 op/s; 0 B/s, 460 objects/s recovering 2024-03-26T21:51:37.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:37 smithi067 ceph-mon[33346]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T21:51:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:37.591 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:37.616 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:37.717 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:37.740 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:37.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:37.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:37.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:37.999 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:38.072 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'unique_pool_4' removed 2024-03-26T21:51:38.102 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:38.156 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:38.189 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:51:38.189 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:51:38.239 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T21:51:38.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:38.258 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:38.281 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:38.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:38.509 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:38.533 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:38.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7 2024-03-26T21:51:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.7' 2024-03-26T21:51:38.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:38.786 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:38.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:38.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:39.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:39.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[40587]: pgmap v1089: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 0 B/s, 393 objects/s recovering 2024-03-26T21:51:39.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:39.065 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[40587]: osdmap e498: 8 total, 8 up, 7 in 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[29406]: pgmap v1089: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 0 B/s, 393 objects/s recovering 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:39 smithi055 ceph-mon[29406]: osdmap e498: 8 total, 8 up, 7 in 2024-03-26T21:51:39.066 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:39.146 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:39.170 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:39.272 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:39.296 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:39.397 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:39.421 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:39 smithi067 ceph-mon[33346]: pgmap v1089: 148 pgs: 1 active+undersized+remapped, 147 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 0 B/s, 393 objects/s recovering 2024-03-26T21:51:39.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:39.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:39 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1048764650' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T21:51:39.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:39 smithi067 ceph-mon[33346]: osdmap e498: 8 total, 8 up, 7 in 2024-03-26T21:51:39.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:39.594 INFO:tasks.thrashosds.thrasher:j is {'epoch': 497, 'fsid': 'e5f52b02-ebb6-11ee-95d0-87774f69a715', 'created': '2024-03-26T21:22:56.673257+0000', 'modified': '2024-03-26T21:51:35.004748+0000', 'last_up_change': '2024-03-26T21:41:24.572964+0000', 'last_in_change': '2024-03-26T21:48:57.950733+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T21:26:13.884991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '484', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T21:28:33.142617+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T21:28:33.591599+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T21:28:34.904126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T21:28:36.852373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T21:28:38.937273+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-26T21:47:37.048063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '439', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '4f487071-c151-406b-b25b-3135cb3c63ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 471, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6801', 'nonce': 317381212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6802', 'nonce': 317381212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6804', 'nonce': 317381212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6803', 'nonce': 317381212}]}, 'public_addr': '172.21.15.55:6801/317381212', 'cluster_addr': '172.21.15.55:6802/317381212', 'heartbeat_back_addr': '172.21.15.55:6804/317381212', 'heartbeat_front_addr': '172.21.15.55:6803/317381212', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4fbc90fe-926a-4ac6-b5f0-d483ba4c09db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6805', 'nonce': 3733554168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6806', 'nonce': 3733554168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6808', 'nonce': 3733554168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6807', 'nonce': 3733554168}]}, 'public_addr': '172.21.15.55:6805/3733554168', 'cluster_addr': '172.21.15.55:6806/3733554168', 'heartbeat_back_addr': '172.21.15.55:6808/3733554168', 'heartbeat_front_addr': '172.21.15.55:6807/3733554168', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e2fe8018-1060-499c-a7e6-56d4da0eba7b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 433, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6809', 'nonce': 3059242923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6810', 'nonce': 3059242923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6812', 'nonce': 3059242923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6811', 'nonce': 3059242923}]}, 'public_addr': '172.21.15.55:6809/3059242923', 'cluster_addr': '172.21.15.55:6810/3059242923', 'heartbeat_back_addr': '172.21.15.55:6812/3059242923', 'heartbeat_front_addr': '172.21.15.55:6811/3059242923', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '30637a8d-5c1b-4ede-8214-a4595c875d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 496, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6813', 'nonce': 3851321468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6814', 'nonce': 3851321468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6816', 'nonce': 3851321468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6815', 'nonce': 3851321468}]}, 'public_addr': '172.21.15.55:6813/3851321468', 'cluster_addr': '172.21.15.55:6814/3851321468', 'heartbeat_back_addr': '172.21.15.55:6816/3851321468', 'heartbeat_front_addr': '172.21.15.55:6815/3851321468', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'cd2d68be-4b10-4242-826b-e085c2ce5598', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 31, 'last_clean_end': 92, 'up_from': 133, 'up_thru': 494, 'down_at': 93, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 254752278}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6801', 'nonce': 254752278}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6803', 'nonce': 254752278}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 254752278}]}, 'public_addr': '172.21.15.67:6800/254752278', 'cluster_addr': '172.21.15.67:6801/254752278', 'heartbeat_back_addr': '172.21.15.67:6803/254752278', 'heartbeat_front_addr': '172.21.15.67:6802/254752278', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7468b94a-3216-4bc2-9b57-1d93e7c0b213', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 467, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 278711358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6805', 'nonce': 278711358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6807', 'nonce': 278711358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 278711358}]}, 'public_addr': '172.21.15.67:6804/278711358', 'cluster_addr': '172.21.15.67:6805/278711358', 'heartbeat_back_addr': '172.21.15.67:6807/278711358', 'heartbeat_front_addr': '172.21.15.67:6806/278711358', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd47c79f3-96bf-46b6-a7f3-a315446d04b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 433, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 3926251639}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 3926251639}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6811', 'nonce': 3926251639}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 3926251639}]}, 'public_addr': '172.21.15.67:6808/3926251639', 'cluster_addr': '172.21.15.67:6809/3926251639', 'heartbeat_back_addr': '172.21.15.67:6811/3926251639', 'heartbeat_front_addr': '172.21.15.67:6810/3926251639', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f10b98b8-887d-4f83-8bc9-99f5c302dc55', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 59, 'last_clean_end': 248, 'up_from': 276, 'up_thru': 476, 'down_at': 249, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 650373445}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6813', 'nonce': 650373445}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 650373445}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 650373445}]}, 'public_addr': '172.21.15.67:6812/650373445', 'cluster_addr': '172.21.15.67:6813/650373445', 'heartbeat_back_addr': '172.21.15.67:6815/650373445', 'heartbeat_front_addr': '172.21.15.67:6814/650373445', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:51:33.212103+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:51:05.267779+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T21:51:19.896579+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:51:09.854116+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:51:29.231740+0000', 'dead_epoch': 92}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:51:30.396887+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:50:50.350586+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T21:50:53.113466+0000', 'dead_epoch': 248}], 'pg_upmap': [{'pgid': '5.17', 'osds': [1, 2, 3]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.55:0/2649740776': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/4141219490': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/147833426': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2854772543': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:6800/2804789175': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/2199694469': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:0/1614234257': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/2834138692': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/1740219825': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/268664696': '2024-03-27T21:23:57.787077+0000', '172.21.15.55:6800/536717527': '2024-03-27T21:23:30.740110+0000', '172.21.15.55:0/966571165': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/2757111431': '2024-03-27T21:29:15.251747+0000', '172.21.15.55:0/559658018': '2024-03-27T21:23:57.787077+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T21:51:39.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:51:39.650 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1 2024-03-26T21:51:39.674 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.1' 2024-03-26T21:51:39.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:39.800 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:39.902 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:39.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:39.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:40.027 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:40.050 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:40.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "size": 20, 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 600, 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "ops": [ 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241352 9.8 9:1a77692d:::benchmark_data_smithi055_135863_object110701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.97880400700001, 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.96528207899999996, 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.161 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241352 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.162 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876450+0000", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876452+0000", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1229999999999998e-06 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876461+0000", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.9670000000000007e-06 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.163 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876481+0000", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.0361e-05 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876493+0000", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.169e-05 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876566+0000", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3020000000000002e-05 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.876576+0000", 2024-03-26T21:51:40.164 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9399999999999997e-06 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836333+0000", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.95975673500000003 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836431+0000", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8459e-05 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.836511+0000", 2024-03-26T21:51:40.165 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0210999999999995e-05 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841566+0000", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0050547559999999997 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841607+0000", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1267999999999999e-05 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841648+0000", 2024-03-26T21:51:40.166 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0703e-05 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841697+0000", 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9001000000000001e-05 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841732+0000", 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4845000000000003e-05 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.167 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241370 9.8 9:1a2077f5:::benchmark_data_smithi055_135863_object110719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.95654189899994, 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94483976700000005, 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241370 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.168 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898712+0000", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898714+0000", 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.621e-06 2024-03-26T21:51:40.169 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898721+0000", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3529999999999997e-06 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898737+0000", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6283e-05 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898747+0000", 2024-03-26T21:51:40.170 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.2979999999999997e-06 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898812+0000", 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.5160000000000006e-05 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.898823+0000", 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0910000000000001e-05 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.171 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838884+0000", 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.94006149800000005 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.838956+0000", 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.2070999999999995e-05 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.839026+0000", 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.9647999999999995e-05 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.172 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841779+0000", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0027533039999999998 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843421+0000", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0016415379999999999 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843462+0000", 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0815999999999997e-05 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.173 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843511+0000", 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9166999999999999e-05 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843552+0000", 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1100000000000003e-05 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.174 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241386 9.8 9:15e63956:::benchmark_data_smithi055_135863_object110735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.94026570899996, 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92876690900000003, 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241386 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.175 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914988+0000", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914989+0000", 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1060000000000001e-06 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.176 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.914994+0000", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.04e-06 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915006+0000", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1435e-05 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915014+0000", 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.7619999999999995e-06 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.177 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915071+0000", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7256000000000002e-05 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.915079+0000", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.3780000000000001e-06 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840207+0000", 2024-03-26T21:51:40.178 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92512746099999998 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840287+0000", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0653000000000002e-05 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.840368+0000", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.0980000000000001e-05 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843430+0000", 2024-03-26T21:51:40.179 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0030614000000000001 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843613+0000", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000182805 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843672+0000", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9079000000000001e-05 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843722+0000", 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0284000000000002e-05 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.180 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843755+0000", 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3269999999999998e-05 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241396 9.8 9:15ab1784:::benchmark_data_smithi055_135863_object110745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.93385551300003, 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.928642615, 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.181 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241396 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.182 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921398+0000", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921399+0000", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.032e-06 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921405+0000", 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.0209999999999999e-06 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.183 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921417+0000", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2874000000000001e-05 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921426+0000", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1429999999999998e-06 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921482+0000", 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.6236999999999997e-05 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.184 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.921490+0000", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9610000000000005e-06 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841812+0000", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.92032255900000004 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.841909+0000", 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.7065999999999995e-05 2024-03-26T21:51:40.185 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.842006+0000", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6650999999999993e-05 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845746+0000", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.003739839 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845805+0000", 2024-03-26T21:51:40.186 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.9500000000000003e-05 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.849965+0000", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0041593410000000004 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850001+0000", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.6341000000000003e-05 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.850041+0000", 2024-03-26T21:51:40.187 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.0049999999999998e-05 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241442 9.8 9:117f2a93:::benchmark_data_smithi055_135863_object110791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.89754766900001, 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90324378800000005, 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.188 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241442 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957706+0000", 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.189 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957708+0000", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.956e-06 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957726+0000", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.8070000000000001e-05 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957742+0000", 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5252000000000001e-05 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.190 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957750+0000", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1359999999999997e-06 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957831+0000", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1278000000000003e-05 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.957843+0000", 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2038999999999999e-05 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.191 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843789+0000", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88594599600000001 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843888+0000", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.8515000000000001e-05 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.843989+0000", 2024-03-26T21:51:40.192 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000101331 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852756+0000", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0087672730000000008 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852772+0000", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5968999999999999e-05 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860849+0000", 2024-03-26T21:51:40.193 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0080765820000000006 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860901+0000", 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.2301e-05 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860950+0000", 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.9089999999999999e-05 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.194 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241445 9.8 9:195f423d:::benchmark_data_smithi055_135863_object110794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.89655388699998, 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.902364579, 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241445 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.195 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958700+0000", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958701+0000", 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.4399999999999999e-07 2024-03-26T21:51:40.196 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958706+0000", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.091e-06 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958721+0000", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4491999999999999e-05 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958729+0000", 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.541e-06 2024-03-26T21:51:40.197 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958812+0000", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.2684999999999999e-05 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.958822+0000", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0095000000000001e-05 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845839+0000", 2024-03-26T21:51:40.198 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88701723600000004 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.845945+0000", 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000105958 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.846050+0000", 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000104899 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.199 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852779+0000", 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0067287370000000003 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.852812+0000", 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.3673000000000001e-05 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860989+0000", 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0081766100000000008 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.200 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861031+0000", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.1711000000000001e-05 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861065+0000", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.4007000000000002e-05 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.25220.0:241482 9.8 9:1973c4a0:::benchmark_data_smithi055_135863_object110831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:51:40.201 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 529.86771688800002, 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.88112820999999997, 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.25220", 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2033055687", 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 241482 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.202 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987537+0000", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987539+0000", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5999999999999999e-06 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.203 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987545+0000", 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.2820000000000002e-06 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987558+0000", 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2744e-05 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987566+0000", 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.1259999999999998e-06 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.204 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987642+0000", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.6149999999999994e-05 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:49.987652+0000", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0351e-05 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847815+0000", 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.86016230900000001 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.205 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847915+0000", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000100707 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 5,7", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.847989+0000", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.3889999999999999e-05 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.860773+0000", 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.012783341 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.206 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.861101+0000", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00032820899999999999 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868358+0000", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0072569499999999999 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868618+0000", 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00025993400000000002 2024-03-26T21:51:40.207 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:42:50.868665+0000", 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.7617000000000002e-05 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:52100 10.8 10:13e4aa23:::benchmark_data_smithi055_163313_object52099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 455.51998682599998, 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.023603611, 2024-03-26T21:51:40.208 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "reached pg", 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 52100 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335267+0000", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335269+0000", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.57e-06 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335299+0000", 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.0700000000000001e-05 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.210 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335311+0000", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1131000000000001e-05 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335318+0000", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9109999999999993e-06 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335557+0000", 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023892 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.211 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.335563+0000", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.8309999999999997e-06 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.463863+0000", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.128299688 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.463877+0000", 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3730999999999999e-05 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.212 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.358791+0000", 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.89491480700000003 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.358871+0000", 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9321999999999997e-05 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:52111 10.4 10:26e68298:::benchmark_data_smithi055_163313_object52110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-26T21:51:40.213 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 455.34283230699998, 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.85519976200000003, 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 52111 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.214 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512422+0000", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512423+0000", 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.6899999999999999e-06 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.215 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512448+0000", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.4522999999999999e-05 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512465+0000", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.7441000000000001e-05 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512476+0000", 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.029e-05 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.216 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512544+0000", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.8879000000000002e-05 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:04.512556+0000", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1127e-05 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.356974+0000", 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84441795200000003 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.217 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.357121+0000", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00014747100000000001 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,4", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.357306+0000", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000184475 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.361403+0000", 2024-03-26T21:51:40.218 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0040972389999999999 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.365943+0000", 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.004539838 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367325+0000", 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001382716 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.219 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367388+0000", 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.3091999999999998e-05 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:44:05.367621+0000", 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00023302899999999999 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164765 10.20 10:055a7cd5:::benchmark_data_smithi055_163313_object60716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 298.89008178, 2024-03-26T21:51:40.220 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91669290199999998, 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164765 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.221 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965172+0000", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965174+0000", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.499e-06 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965180+0000", 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.0190000000000003e-06 2024-03-26T21:51:40.222 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965192+0000", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2476999999999999e-05 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965201+0000", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0960000000000007e-06 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965298+0000", 2024-03-26T21:51:40.223 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6576999999999994e-05 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.965309+0000", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1460999999999999e-05 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877721+0000", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.91241161400000004 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877812+0000", 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1153e-05 2024-03-26T21:51:40.224 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.877996+0000", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00018388599999999999 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881439+0000", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0034433939999999998 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881533+0000", 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.3682000000000002e-05 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.225 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881763+0000", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000229497 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881821+0000", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.7992999999999999e-05 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881865+0000", 2024-03-26T21:51:40.226 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4554000000000001e-05 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164781 10.20 10:0451c1db:::benchmark_data_smithi055_163313_object60732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 298.877908252, 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90551289400000001, 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.227 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164781 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977346+0000", 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.228 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977347+0000", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.3689999999999999e-06 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977352+0000", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4139999999999996e-06 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977361+0000", 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.1390000000000001e-06 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.229 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977367+0000", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.7159999999999997e-06 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977845+0000", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00047740400000000001 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:40.977855+0000", 2024-03-26T21:51:40.230 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0068000000000001e-05 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879295+0000", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.90144047199999999 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879405+0000", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000110192 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.879593+0000", 2024-03-26T21:51:40.231 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000187982 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881647+0000", 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0020535129999999999 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881694+0000", 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.6876999999999999e-05 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.232 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882794+0000", 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001100209 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882830+0000", 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 3.5611000000000002e-05 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882859+0000", 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.8928000000000002e-05 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.233 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164858 10.20 10:05bf36b7:::benchmark_data_smithi055_163313_object60809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 298.824009569, 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.85176363099999997, 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164858 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.234 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031244+0000", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031246+0000", 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.522e-06 2024-03-26T21:51:40.235 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031253+0000", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6549999999999998e-06 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031274+0000", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.1809000000000001e-05 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031283+0000", 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 8.7630000000000006e-06 2024-03-26T21:51:40.236 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031350+0000", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6995000000000005e-05 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.031360+0000", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.6830000000000003e-06 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880403+0000", 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84904316999999996 2024-03-26T21:51:40.237 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880515+0000", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000112048 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.880682+0000", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000167128 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881880+0000", 2024-03-26T21:51:40.238 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.001197353 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882909+0000", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0010296680000000001 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882957+0000", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.8137999999999999e-05 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882985+0000", 2024-03-26T21:51:40.239 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.758e-05 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.883008+0000", 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3119e-05 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.15609.0:164882 10.20 10:06434657:::benchmark_data_smithi055_163313_object60833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 298.809496153, 2024-03-26T21:51:40.240 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.84022213000000001, 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.15609", 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/2300774870", 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 164882 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.241 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045758+0000", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045759+0000", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.0210000000000001e-06 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045764+0000", 2024-03-26T21:51:40.242 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.327e-06 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045774+0000", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.9920000000000002e-06 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045782+0000", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.9799999999999998e-06 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045984+0000", 2024-03-26T21:51:40.243 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00020188699999999999 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for peered", 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.045995+0000", 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.1035e-05 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.881926+0000", 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.83593075800000005 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.244 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882029+0000", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000103244 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "waiting for subops from 1,2", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.882173+0000", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000144365 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op_commit", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.883023+0000", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00084954200000000001 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.245 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885833+0000", 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0028101329999999998 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "sub_op_commit_rec", 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885884+0000", 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1007999999999997e-05 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "commit_sent", 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885936+0000", 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 5.1517000000000002e-05 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.246 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:46:41.885980+0000", 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 4.4320999999999998e-05 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:4575 5.3 5:c609908c:::notify.5:head [watch ping cookie 94208173337984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e494)", 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:51:30.165294+0000", 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 9.6899598089999994, 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0027719369999999999, 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.247 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 4575 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165294+0000", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.248 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165294+0000", 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165296+0000", 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.9130000000000001e-06 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165306+0000", 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.5629999999999998e-06 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.249 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165329+0000", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.3037999999999999e-05 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165341+0000", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.2534999999999999e-05 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165408+0000", 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 6.6662999999999995e-05 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.250 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "op must wait for map", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.165415+0000", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.1609999999999997e-06 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.167868+0000", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.0024531399999999999 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.167946+0000", 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 7.8003000000000005e-05 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.251 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:30.168066+0000", 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00011992100000000001 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "description": "osd_op(client.14454.0:4585 5.3 5:c609908c:::notify.5:head [watch ping cookie 94208173337984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e495)", 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "initiated_at": "2024-03-26T21:51:35.166342+0000", 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "age": 4.6889121849999995, 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.00044468900000000001, 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "continuous": false, 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "type_data": { 2024-03-26T21:51:40.252 INFO:teuthology.orchestra.run.smithi055.stdout: "flag_point": "started", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "client_info": { 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "client": "client.14454", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "client_addr": "172.21.15.55:0/1460561447", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "tid": 4585 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "events": [ 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "initiated", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166342+0000", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "header_read", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166342+0000", 2024-03-26T21:51:40.253 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "throttled", 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166344+0000", 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.6809999999999998e-06 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "all_read", 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166359+0000", 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.4797000000000001e-05 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "dispatched", 2024-03-26T21:51:40.254 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166389+0000", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 2.9708000000000001e-05 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "queued_for_pg", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166405+0000", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 1.5685e-05 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "reached_pg", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166576+0000", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000171724 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "started", 2024-03-26T21:51:40.255 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166666+0000", 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 9.0037999999999994e-05 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: { 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: "event": "done", 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: "time": "2024-03-26T21:51:35.166786+0000", 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: "duration": 0.000120056 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-03-26T21:51:40.256 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-03-26T21:51:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.257 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:40.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:40.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:40.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:40.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4 2024-03-26T21:51:40.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:40 smithi067 ceph-mon[33346]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T21:51:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.518 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:40.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.4' 2024-03-26T21:51:40.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:40 smithi055 ceph-mon[29406]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T21:51:40.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:40 smithi055 ceph-mon[40587]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T21:51:40.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6 2024-03-26T21:51:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.6' 2024-03-26T21:51:40.748 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3 2024-03-26T21:51:40.771 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.3' 2024-03-26T21:51:40.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:40.897 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:40.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T21:51:40.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5 2024-03-26T21:51:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: docker: command not found 2024-03-26T21:51:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:41.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi067 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.5' 2024-03-26T21:51:41.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2 2024-03-26T21:51:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:41.183 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:41.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.2' 2024-03-26T21:51:41.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:41.311 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:41.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:41.315 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:41 smithi055 ceph-mon[29406]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:51:41.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:41 smithi055 ceph-mon[40587]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:51:41.391 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:41 smithi067 ceph-mon[33346]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:51:41.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0 2024-03-26T21:51:41.434 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-26T21:51:41.445 INFO:tasks.thrashosds:joining thrashosds 2024-03-26T21:51:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T21:51:41.446 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: docker: command not found 2024-03-26T21:51:41.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi055 with status 1: 'sudo docker kill -s 1 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd.0' 2024-03-26T21:51:42.848 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0: {} 2024-03-26T21:51:42.867 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1: {} 2024-03-26T21:51:42.886 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2: {} 2024-03-26T21:51:42.906 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3: {} 2024-03-26T21:51:42.926 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4: {} 2024-03-26T21:51:42.947 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5: {} 2024-03-26T21:51:42.968 INFO:teuthology.orchestra.run.smithi055.stdout:osd.6: {} 2024-03-26T21:51:42.988 INFO:teuthology.orchestra.run.smithi055.stdout:osd.7: {} 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T21:51:42.990 INFO:teuthology.orchestra.run.smithi055.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T21:51:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:43 smithi055 ceph-mon[40587]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 525 B/s rd, 0 op/s 2024-03-26T21:51:43.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:43 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2116337918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:43 smithi055 ceph-mon[29406]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 525 B/s rd, 0 op/s 2024-03-26T21:51:43.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:43 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2116337918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:43.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:43 smithi067 ceph-mon[33346]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 525 B/s rd, 0 op/s 2024-03-26T21:51:43.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:43 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2116337918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T21:51:44.147 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:51:44.405 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:44.430 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:44.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:44 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1583862935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:44.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:44 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:44.530 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:44 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1583862935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:44.530 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:44 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:44.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:44 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1583862935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:44.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:44 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:45.457 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[29406]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:45.457 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[29406]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:45.457 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:45.457 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[29406]: osdmap e499: 8 total, 8 up, 7 in 2024-03-26T21:51:45.458 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[40587]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:45.458 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[40587]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:45.458 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:45.458 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:45 smithi055 ceph-mon[40587]: osdmap e499: 8 total, 8 up, 7 in 2024-03-26T21:51:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:45 smithi067 ceph-mon[33346]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:45.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:45 smithi067 ceph-mon[33346]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T21:51:45.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:45.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:45 smithi067 ceph-mon[33346]: osdmap e499: 8 total, 8 up, 7 in 2024-03-26T21:51:46.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T21:51:47.202 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 4.1d scrub starts 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 4.1d scrub ok 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:51:47.203 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 4.1d scrub starts 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 4.1d scrub ok 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:51:47.204 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:47 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:51:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:51:47.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 4.1d scrub starts 2024-03-26T21:51:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 4.1d scrub ok 2024-03-26T21:51:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:51:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:51:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:51:47.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:51:47.991 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:51:47.991 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:51:48.073 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:51:48.075 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:51:48.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:48.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:51:48.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:48.547 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:51:48.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:51:48.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T21:51:49.148 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:49.328 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1096,"stamp":"2024-03-26T21:51:47.800677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3009,"num_read_kb":2857,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1778906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1317,"ondisk_log_size":1317,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5775860,"kb_used_data":7456,"kb_used_omap":18,"kb_used_meta":5768301,"kb_avail":650440204,"statfs":{"total":671965249536,"available":666050768896,"internally_reserved":0,"allocated":7634944,"data_stored":4708568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19289,"internal_metadata":5906740391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.734516"},"pg_stats":[{"pgid":"4.c","version":"456'35","reported_seq":733,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378589+0000","last_change":"2024-03-26T21:48:03.604234+0000","last_active":"2024-03-26T21:51:44.378589+0000","last_peered":"2024-03-26T21:51:44.378589+0000","last_clean":"2024-03-26T21:51:44.378589+0000","last_became_active":"2024-03-26T21:44:49.410968+0000","last_became_peered":"2024-03-26T21:44:49.410968+0000","last_unstale":"2024-03-26T21:51:44.378589+0000","last_undegraded":"2024-03-26T21:51:44.378589+0000","last_fullsized":"2024-03-26T21:51:44.378589+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:03.603829+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.603829+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068607+0000","last_change":"2024-03-26T21:47:23.290957+0000","last_active":"2024-03-26T21:51:38.068607+0000","last_peered":"2024-03-26T21:51:38.068607+0000","last_clean":"2024-03-26T21:51:38.068607+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:51:38.068607+0000","last_undegraded":"2024-03-26T21:51:38.068607+0000","last_fullsized":"2024-03-26T21:51:38.068607+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:23.290864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.290864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"462'12","reported_seq":764,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066935+0000","last_change":"2024-03-26T21:48:59.006658+0000","last_active":"2024-03-26T21:51:38.066935+0000","last_peered":"2024-03-26T21:51:38.066935+0000","last_clean":"2024-03-26T21:51:38.066935+0000","last_became_active":"2024-03-26T21:48:58.995625+0000","last_became_peered":"2024-03-26T21:48:58.995625+0000","last_unstale":"2024-03-26T21:51:38.066935+0000","last_undegraded":"2024-03-26T21:51:38.066935+0000","last_fullsized":"2024-03-26T21:51:38.066935+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-03-26T21:48:03.065913+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.065913+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":643,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379717+0000","last_change":"2024-03-26T21:48:58.976822+0000","last_active":"2024-03-26T21:51:44.379717+0000","last_peered":"2024-03-26T21:51:44.379717+0000","last_clean":"2024-03-26T21:51:44.379717+0000","last_became_active":"2024-03-26T21:48:58.976172+0000","last_became_peered":"2024-03-26T21:48:58.976172+0000","last_unstale":"2024-03-26T21:51:44.379717+0000","last_undegraded":"2024-03-26T21:51:44.379717+0000","last_fullsized":"2024-03-26T21:51:44.379717+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.630764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.630764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760519+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":729,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378598+0000","last_change":"2024-03-26T21:48:59.000246+0000","last_active":"2024-03-26T21:51:44.378598+0000","last_peered":"2024-03-26T21:51:44.378598+0000","last_clean":"2024-03-26T21:51:44.378598+0000","last_became_active":"2024-03-26T21:48:58.980284+0000","last_became_peered":"2024-03-26T21:48:58.980284+0000","last_unstale":"2024-03-26T21:51:44.378598+0000","last_undegraded":"2024-03-26T21:51:44.378598+0000","last_fullsized":"2024-03-26T21:51:44.378598+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'39","last_scrub_stamp":"2024-03-26T21:47:23.549927+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.549927+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":401,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379123+0000","last_change":"2024-03-26T21:49:29.974892+0000","last_active":"2024-03-26T21:51:44.379123+0000","last_peered":"2024-03-26T21:51:44.379123+0000","last_clean":"2024-03-26T21:51:44.379123+0000","last_became_active":"2024-03-26T21:49:29.974562+0000","last_became_peered":"2024-03-26T21:49:29.974562+0000","last_unstale":"2024-03-26T21:51:44.379123+0000","last_undegraded":"2024-03-26T21:51:44.379123+0000","last_fullsized":"2024-03-26T21:51:44.379123+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:24.976075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:47:24.976075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":968,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.082046+0000","last_change":"2024-03-26T21:48:01.699674+0000","last_active":"2024-03-26T21:51:38.082046+0000","last_peered":"2024-03-26T21:51:38.082046+0000","last_clean":"2024-03-26T21:51:38.082046+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:51:38.082046+0000","last_undegraded":"2024-03-26T21:51:38.082046+0000","last_fullsized":"2024-03-26T21:51:38.082046+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:01.699433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.699433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":765,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069368+0000","last_change":"2024-03-26T21:48:58.974425+0000","last_active":"2024-03-26T21:51:38.069368+0000","last_peered":"2024-03-26T21:51:38.069368+0000","last_clean":"2024-03-26T21:51:38.069368+0000","last_became_active":"2024-03-26T21:48:58.974080+0000","last_became_peered":"2024-03-26T21:48:58.974080+0000","last_unstale":"2024-03-26T21:51:38.069368+0000","last_undegraded":"2024-03-26T21:51:38.069368+0000","last_fullsized":"2024-03-26T21:51:38.069368+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.289907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:04.923329+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.289907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997572+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":678,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379863+0000","last_change":"2024-03-26T21:47:27.572650+0000","last_active":"2024-03-26T21:51:44.379863+0000","last_peered":"2024-03-26T21:51:44.379863+0000","last_clean":"2024-03-26T21:51:44.379863+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:51:44.379863+0000","last_undegraded":"2024-03-26T21:51:44.379863+0000","last_fullsized":"2024-03-26T21:51:44.379863+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"426'24","last_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:47:27.572552+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":652,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380101+0000","last_change":"2024-03-26T21:48:58.979871+0000","last_active":"2024-03-26T21:51:44.380101+0000","last_peered":"2024-03-26T21:51:44.380101+0000","last_clean":"2024-03-26T21:51:44.380101+0000","last_became_active":"2024-03-26T21:48:58.979258+0000","last_became_peered":"2024-03-26T21:48:58.979258+0000","last_unstale":"2024-03-26T21:51:44.380101+0000","last_undegraded":"2024-03-26T21:51:44.380101+0000","last_fullsized":"2024-03-26T21:51:44.380101+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.750677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":968,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378945+0000","last_change":"2024-03-26T21:50:51.868540+0000","last_active":"2024-03-26T21:51:44.378945+0000","last_peered":"2024-03-26T21:51:44.378945+0000","last_clean":"2024-03-26T21:51:44.378945+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:51:44.378945+0000","last_undegraded":"2024-03-26T21:51:44.378945+0000","last_fullsized":"2024-03-26T21:51:44.378945+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:51.868407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:50:51.868407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:52:09.161322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.070193+0000","last_change":"2024-03-26T21:48:30.759102+0000","last_active":"2024-03-26T21:51:38.070193+0000","last_peered":"2024-03-26T21:51:38.070193+0000","last_clean":"2024-03-26T21:51:38.070193+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:51:38.070193+0000","last_undegraded":"2024-03-26T21:51:38.070193+0000","last_fullsized":"2024-03-26T21:51:38.070193+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.758986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.758986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997587+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1105,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.089598+0000","last_change":"2024-03-26T21:48:08.701995+0000","last_active":"2024-03-26T21:51:38.089598+0000","last_peered":"2024-03-26T21:51:38.089598+0000","last_clean":"2024-03-26T21:51:38.089598+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:51:38.089598+0000","last_undegraded":"2024-03-26T21:51:38.089598+0000","last_fullsized":"2024-03-26T21:51:38.089598+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:08.701304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.701304+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161428+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067599+0000","last_change":"2024-03-26T21:48:21.815329+0000","last_active":"2024-03-26T21:51:38.067599+0000","last_peered":"2024-03-26T21:51:38.067599+0000","last_clean":"2024-03-26T21:51:38.067599+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:51:38.067599+0000","last_undegraded":"2024-03-26T21:51:38.067599+0000","last_fullsized":"2024-03-26T21:51:38.067599+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.815090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.815090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997521+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"205'12","reported_seq":829,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380012+0000","last_change":"2024-03-26T21:48:25.705142+0000","last_active":"2024-03-26T21:51:44.380012+0000","last_peered":"2024-03-26T21:51:44.380012+0000","last_clean":"2024-03-26T21:51:44.380012+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:51:44.380012+0000","last_undegraded":"2024-03-26T21:51:44.380012+0000","last_fullsized":"2024-03-26T21:51:44.380012+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'12","last_scrub_stamp":"2024-03-26T21:48:25.705058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:25.705058+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":943,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.377889+0000","last_change":"2024-03-26T21:50:01.887630+0000","last_active":"2024-03-26T21:51:44.377889+0000","last_peered":"2024-03-26T21:51:44.377889+0000","last_clean":"2024-03-26T21:51:44.377889+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:51:44.377889+0000","last_undegraded":"2024-03-26T21:51:44.377889+0000","last_fullsized":"2024-03-26T21:51:44.377889+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:01.887146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:50:01.887146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703640+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":657,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380242+0000","last_change":"2024-03-26T21:48:15.723533+0000","last_active":"2024-03-26T21:51:44.380242+0000","last_peered":"2024-03-26T21:51:44.380242+0000","last_clean":"2024-03-26T21:51:44.380242+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:51:44.380242+0000","last_undegraded":"2024-03-26T21:51:44.380242+0000","last_fullsized":"2024-03-26T21:51:44.380242+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:15.723423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.872296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.723423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":326,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066902+0000","last_change":"2024-03-26T21:48:01.795766+0000","last_active":"2024-03-26T21:51:38.066902+0000","last_peered":"2024-03-26T21:51:38.066902+0000","last_clean":"2024-03-26T21:51:38.066902+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:51:38.066902+0000","last_undegraded":"2024-03-26T21:51:38.066902+0000","last_fullsized":"2024-03-26T21:51:38.066902+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'38","last_scrub_stamp":"2024-03-26T21:48:01.795624+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.795624+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997556+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069763+0000","last_change":"2024-03-26T21:48:09.725502+0000","last_active":"2024-03-26T21:51:38.069763+0000","last_peered":"2024-03-26T21:51:38.069763+0000","last_clean":"2024-03-26T21:51:38.069763+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:51:38.069763+0000","last_undegraded":"2024-03-26T21:51:38.069763+0000","last_fullsized":"2024-03-26T21:51:38.069763+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:09.725378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.725378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":664,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379561+0000","last_change":"2024-03-26T21:48:58.977779+0000","last_active":"2024-03-26T21:51:44.379561+0000","last_peered":"2024-03-26T21:51:44.379561+0000","last_clean":"2024-03-26T21:51:44.379561+0000","last_became_active":"2024-03-26T21:48:58.977356+0000","last_became_peered":"2024-03-26T21:48:58.977356+0000","last_unstale":"2024-03-26T21:51:44.379561+0000","last_undegraded":"2024-03-26T21:51:44.379561+0000","last_fullsized":"2024-03-26T21:51:44.379561+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:28.651656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:48:28.651656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":377,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067750+0000","last_change":"2024-03-26T21:47:11.403625+0000","last_active":"2024-03-26T21:51:38.067750+0000","last_peered":"2024-03-26T21:51:38.067750+0000","last_clean":"2024-03-26T21:51:38.067750+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:51:38.067750+0000","last_undegraded":"2024-03-26T21:51:38.067750+0000","last_fullsized":"2024-03-26T21:51:38.067750+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:47:11.403511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:11.403511+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.024673+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":696,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380117+0000","last_change":"2024-03-26T21:47:26.605446+0000","last_active":"2024-03-26T21:51:44.380117+0000","last_peered":"2024-03-26T21:51:44.380117+0000","last_clean":"2024-03-26T21:51:44.380117+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:51:44.380117+0000","last_undegraded":"2024-03-26T21:51:44.380117+0000","last_fullsized":"2024-03-26T21:51:44.380117+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'28","last_scrub_stamp":"2024-03-26T21:47:26.605347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:26.605347+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":953,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023003+0000","last_change":"2024-03-26T21:48:17.934161+0000","last_active":"2024-03-26T21:51:39.023003+0000","last_peered":"2024-03-26T21:51:39.023003+0000","last_clean":"2024-03-26T21:51:39.023003+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:51:39.023003+0000","last_undegraded":"2024-03-26T21:51:39.023003+0000","last_fullsized":"2024-03-26T21:51:39.023003+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-26T21:48:17.933856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.933856+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":864,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066116+0000","last_change":"2024-03-26T21:48:58.986678+0000","last_active":"2024-03-26T21:51:38.066116+0000","last_peered":"2024-03-26T21:51:38.066116+0000","last_clean":"2024-03-26T21:51:38.066116+0000","last_became_active":"2024-03-26T21:48:58.985775+0000","last_became_peered":"2024-03-26T21:48:58.985775+0000","last_unstale":"2024-03-26T21:51:38.066116+0000","last_undegraded":"2024-03-26T21:51:38.066116+0000","last_fullsized":"2024-03-26T21:51:38.066116+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.992360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.992360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227998+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1218,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.137794+0000","last_change":"2024-03-26T21:48:32.957198+0000","last_active":"2024-03-26T21:51:39.137794+0000","last_peered":"2024-03-26T21:51:39.137794+0000","last_clean":"2024-03-26T21:51:39.137794+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:51:39.137794+0000","last_undegraded":"2024-03-26T21:51:39.137794+0000","last_fullsized":"2024-03-26T21:51:39.137794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:48:32.957059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.957059+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":1034,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.081914+0000","last_change":"2024-03-26T21:48:16.764953+0000","last_active":"2024-03-26T21:51:38.081914+0000","last_peered":"2024-03-26T21:51:38.081914+0000","last_clean":"2024-03-26T21:51:38.081914+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:51:38.081914+0000","last_undegraded":"2024-03-26T21:51:38.081914+0000","last_fullsized":"2024-03-26T21:51:38.081914+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:16.764813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.764813+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161409+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":982,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378232+0000","last_change":"2024-03-26T21:50:46.848428+0000","last_active":"2024-03-26T21:51:44.378232+0000","last_peered":"2024-03-26T21:51:44.378232+0000","last_clean":"2024-03-26T21:51:44.378232+0000","last_became_active":"2024-03-26T21:48:58.979760+0000","last_became_peered":"2024-03-26T21:48:58.979760+0000","last_unstale":"2024-03-26T21:51:44.378232+0000","last_undegraded":"2024-03-26T21:51:44.378232+0000","last_fullsized":"2024-03-26T21:51:44.378232+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:46.848145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:50:46.848145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:52:16.452933+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":946,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.021464+0000","last_change":"2024-03-26T21:48:58.973098+0000","last_active":"2024-03-26T21:51:39.021464+0000","last_peered":"2024-03-26T21:51:39.021464+0000","last_clean":"2024-03-26T21:51:39.021464+0000","last_became_active":"2024-03-26T21:48:58.972760+0000","last_became_peered":"2024-03-26T21:48:58.972760+0000","last_unstale":"2024-03-26T21:51:39.021464+0000","last_undegraded":"2024-03-26T21:51:39.021464+0000","last_fullsized":"2024-03-26T21:51:39.021464+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:30.031216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.031216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":73,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022845+0000","last_change":"2024-03-26T21:49:29.979212+0000","last_active":"2024-03-26T21:51:39.022845+0000","last_peered":"2024-03-26T21:51:39.022845+0000","last_clean":"2024-03-26T21:51:39.022845+0000","last_became_active":"2024-03-26T21:49:29.978839+0000","last_became_peered":"2024-03-26T21:49:29.978839+0000","last_unstale":"2024-03-26T21:51:39.022845+0000","last_undegraded":"2024-03-26T21:51:39.022845+0000","last_fullsized":"2024-03-26T21:51:39.022845+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.087588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.087588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":1011,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022909+0000","last_change":"2024-03-26T21:48:35.997876+0000","last_active":"2024-03-26T21:51:39.022909+0000","last_peered":"2024-03-26T21:51:39.022909+0000","last_clean":"2024-03-26T21:51:39.022909+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:51:39.022909+0000","last_undegraded":"2024-03-26T21:51:39.022909+0000","last_fullsized":"2024-03-26T21:51:39.022909+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'29","last_scrub_stamp":"2024-03-26T21:48:35.997722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.997722+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":970,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.081932+0000","last_change":"2024-03-26T21:48:58.981784+0000","last_active":"2024-03-26T21:51:38.081932+0000","last_peered":"2024-03-26T21:51:38.081932+0000","last_clean":"2024-03-26T21:51:38.081932+0000","last_became_active":"2024-03-26T21:48:58.981168+0000","last_became_peered":"2024-03-26T21:48:58.981168+0000","last_unstale":"2024-03-26T21:51:38.081932+0000","last_undegraded":"2024-03-26T21:51:38.081932+0000","last_fullsized":"2024-03-26T21:51:38.081932+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:19.764889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.764889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161450+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":950,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.075836+0000","last_change":"2024-03-26T21:48:58.983664+0000","last_active":"2024-03-26T21:51:38.075836+0000","last_peered":"2024-03-26T21:51:38.075836+0000","last_clean":"2024-03-26T21:51:38.075836+0000","last_became_active":"2024-03-26T21:48:58.983312+0000","last_became_peered":"2024-03-26T21:48:58.983312+0000","last_unstale":"2024-03-26T21:51:38.075836+0000","last_undegraded":"2024-03-26T21:51:38.075836+0000","last_fullsized":"2024-03-26T21:51:38.075836+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:40.809302+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.809302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161548+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":947,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379074+0000","last_change":"2024-03-26T21:49:30.726913+0000","last_active":"2024-03-26T21:51:44.379074+0000","last_peered":"2024-03-26T21:51:44.379074+0000","last_clean":"2024-03-26T21:51:44.379074+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:51:44.379074+0000","last_undegraded":"2024-03-26T21:51:44.379074+0000","last_fullsized":"2024-03-26T21:51:44.379074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:30.726780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:30.726780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703349+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":1029,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022887+0000","last_change":"2024-03-26T21:48:59.007492+0000","last_active":"2024-03-26T21:51:39.022887+0000","last_peered":"2024-03-26T21:51:39.022887+0000","last_clean":"2024-03-26T21:51:39.022887+0000","last_became_active":"2024-03-26T21:48:58.986240+0000","last_became_peered":"2024-03-26T21:48:58.986240+0000","last_unstale":"2024-03-26T21:51:39.022887+0000","last_undegraded":"2024-03-26T21:51:39.022887+0000","last_fullsized":"2024-03-26T21:51:39.022887+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'24","last_scrub_stamp":"2024-03-26T21:48:11.911818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.911818+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2181,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:41.807281+0000","last_change":"2024-03-26T21:48:19.889390+0000","last_active":"2024-03-26T21:51:41.807281+0000","last_peered":"2024-03-26T21:51:41.807281+0000","last_clean":"2024-03-26T21:51:41.807281+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:51:41.807281+0000","last_undegraded":"2024-03-26T21:51:41.807281+0000","last_fullsized":"2024-03-26T21:51:41.807281+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:48:19.889302+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:39:33.064714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:19.889302+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:47.234799+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1267,"num_read_kb":1262,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":973,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022584+0000","last_change":"2024-03-26T21:48:41.039131+0000","last_active":"2024-03-26T21:51:39.022584+0000","last_peered":"2024-03-26T21:51:39.022584+0000","last_clean":"2024-03-26T21:51:39.022584+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:51:39.022584+0000","last_undegraded":"2024-03-26T21:51:39.022584+0000","last_fullsized":"2024-03-26T21:51:39.022584+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:41.035559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:41.035559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":956,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.074812+0000","last_change":"2024-03-26T21:47:28.333838+0000","last_active":"2024-03-26T21:51:38.074812+0000","last_peered":"2024-03-26T21:51:38.074812+0000","last_clean":"2024-03-26T21:51:38.074812+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:51:38.074812+0000","last_undegraded":"2024-03-26T21:51:38.074812+0000","last_fullsized":"2024-03-26T21:51:38.074812+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:28.333756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:28.333756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":881,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066523+0000","last_change":"2024-03-26T21:48:58.992875+0000","last_active":"2024-03-26T21:51:38.066523+0000","last_peered":"2024-03-26T21:51:38.066523+0000","last_clean":"2024-03-26T21:51:38.066523+0000","last_became_active":"2024-03-26T21:48:58.992117+0000","last_became_peered":"2024-03-26T21:48:58.992117+0000","last_unstale":"2024-03-26T21:51:38.066523+0000","last_undegraded":"2024-03-26T21:51:38.066523+0000","last_fullsized":"2024-03-26T21:51:38.066523+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.882422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.882422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"468'55","reported_seq":1347,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378421+0000","last_change":"2024-03-26T21:49:29.985516+0000","last_active":"2024-03-26T21:51:44.378421+0000","last_peered":"2024-03-26T21:51:44.378421+0000","last_clean":"2024-03-26T21:51:44.378421+0000","last_became_active":"2024-03-26T21:49:29.985117+0000","last_became_peered":"2024-03-26T21:49:29.985117+0000","last_unstale":"2024-03-26T21:51:44.378421+0000","last_undegraded":"2024-03-26T21:51:44.378421+0000","last_fullsized":"2024-03-26T21:51:44.378421+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'42","last_scrub_stamp":"2024-03-26T21:48:23.427209+0000","last_deep_scrub":"277'42","last_deep_scrub_stamp":"2024-03-26T21:41:35.106815+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.427209+0000","objects_scrubbed":8,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760743+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.078363+0000","last_change":"2024-03-26T21:47:13.330107+0000","last_active":"2024-03-26T21:51:38.078363+0000","last_peered":"2024-03-26T21:51:38.078363+0000","last_clean":"2024-03-26T21:51:38.078363+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:51:38.078363+0000","last_undegraded":"2024-03-26T21:51:38.078363+0000","last_fullsized":"2024-03-26T21:51:38.078363+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:13.329967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:47:13.329967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997273+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1145,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065699+0000","last_change":"2024-03-26T21:48:27.821566+0000","last_active":"2024-03-26T21:51:38.065699+0000","last_peered":"2024-03-26T21:51:38.065699+0000","last_clean":"2024-03-26T21:51:38.065699+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:51:38.065699+0000","last_undegraded":"2024-03-26T21:51:38.065699+0000","last_fullsized":"2024-03-26T21:51:38.065699+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:48:27.821430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.821430+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.228066+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":976,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378694+0000","last_change":"2024-03-26T21:50:54.034876+0000","last_active":"2024-03-26T21:51:44.378694+0000","last_peered":"2024-03-26T21:51:44.378694+0000","last_clean":"2024-03-26T21:51:44.378694+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:51:44.378694+0000","last_undegraded":"2024-03-26T21:51:44.378694+0000","last_fullsized":"2024-03-26T21:51:44.378694+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:54.034681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:50:54.034681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:52:21.860735+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":571,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378096+0000","last_change":"2024-03-26T21:50:44.855537+0000","last_active":"2024-03-26T21:51:44.378096+0000","last_peered":"2024-03-26T21:51:44.378096+0000","last_clean":"2024-03-26T21:51:44.378096+0000","last_became_active":"2024-03-26T21:48:58.985930+0000","last_became_peered":"2024-03-26T21:48:58.985930+0000","last_unstale":"2024-03-26T21:51:44.378096+0000","last_undegraded":"2024-03-26T21:51:44.378096+0000","last_fullsized":"2024-03-26T21:51:44.378096+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:44.855404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:50:44.855404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:52:11.833213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1003,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.089758+0000","last_change":"2024-03-26T21:48:35.767934+0000","last_active":"2024-03-26T21:51:38.089758+0000","last_peered":"2024-03-26T21:51:38.089758+0000","last_clean":"2024-03-26T21:51:38.089758+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:51:38.089758+0000","last_undegraded":"2024-03-26T21:51:38.089758+0000","last_fullsized":"2024-03-26T21:51:38.089758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'22","last_scrub_stamp":"2024-03-26T21:48:35.767034+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.767034+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161532+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":771,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.077341+0000","last_change":"2024-03-26T21:48:58.978047+0000","last_active":"2024-03-26T21:51:38.077341+0000","last_peered":"2024-03-26T21:51:38.077341+0000","last_clean":"2024-03-26T21:51:38.077341+0000","last_became_active":"2024-03-26T21:48:58.976535+0000","last_became_peered":"2024-03-26T21:48:58.976535+0000","last_unstale":"2024-03-26T21:51:38.077341+0000","last_undegraded":"2024-03-26T21:51:38.077341+0000","last_fullsized":"2024-03-26T21:51:38.077341+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:17.179980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:29.929124+0000","last_clean_scrub_stamp":"2024-03-26T21:48:17.179980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":970,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.071279+0000","last_change":"2024-03-26T21:48:58.982624+0000","last_active":"2024-03-26T21:51:38.071279+0000","last_peered":"2024-03-26T21:51:38.071279+0000","last_clean":"2024-03-26T21:51:38.071279+0000","last_became_active":"2024-03-26T21:48:58.982052+0000","last_became_peered":"2024-03-26T21:48:58.982052+0000","last_unstale":"2024-03-26T21:51:38.071279+0000","last_undegraded":"2024-03-26T21:51:38.071279+0000","last_fullsized":"2024-03-26T21:51:38.071279+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.696017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.696017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":863,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067053+0000","last_change":"2024-03-26T21:48:31.817091+0000","last_active":"2024-03-26T21:51:38.067053+0000","last_peered":"2024-03-26T21:51:38.067053+0000","last_clean":"2024-03-26T21:51:38.067053+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:51:38.067053+0000","last_undegraded":"2024-03-26T21:51:38.067053+0000","last_fullsized":"2024-03-26T21:51:38.067053+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:31.817002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:48:31.817002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.228083+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":953,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022605+0000","last_change":"2024-03-26T21:48:32.000228+0000","last_active":"2024-03-26T21:51:39.022605+0000","last_peered":"2024-03-26T21:51:39.022605+0000","last_clean":"2024-03-26T21:51:39.022605+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:51:39.022605+0000","last_undegraded":"2024-03-26T21:51:39.022605+0000","last_fullsized":"2024-03-26T21:51:39.022605+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.000079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.361458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"462'56","reported_seq":1354,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379148+0000","last_change":"2024-03-26T21:49:57.671639+0000","last_active":"2024-03-26T21:51:44.379148+0000","last_peered":"2024-03-26T21:51:44.379148+0000","last_clean":"2024-03-26T21:51:44.379148+0000","last_became_active":"2024-03-26T21:48:58.981458+0000","last_became_peered":"2024-03-26T21:48:58.981458+0000","last_unstale":"2024-03-26T21:51:44.379148+0000","last_undegraded":"2024-03-26T21:51:44.379148+0000","last_fullsized":"2024-03-26T21:51:44.379148+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'56","last_scrub_stamp":"2024-03-26T21:49:57.671485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:49:57.671485+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":86,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378175+0000","last_change":"2024-03-26T21:48:58.976086+0000","last_active":"2024-03-26T21:51:44.378175+0000","last_peered":"2024-03-26T21:51:44.378175+0000","last_clean":"2024-03-26T21:51:44.378175+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:51:44.378175+0000","last_undegraded":"2024-03-26T21:51:44.378175+0000","last_fullsized":"2024-03-26T21:51:44.378175+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.160883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:25.349420+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.160883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":974,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378402+0000","last_change":"2024-03-26T21:50:42.898947+0000","last_active":"2024-03-26T21:51:44.378402+0000","last_peered":"2024-03-26T21:51:44.378402+0000","last_clean":"2024-03-26T21:51:44.378402+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:51:44.378402+0000","last_undegraded":"2024-03-26T21:51:44.378402+0000","last_fullsized":"2024-03-26T21:51:44.378402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:50:42.898428+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:50:42.898428+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:49.293056+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069168+0000","last_change":"2024-03-26T21:48:29.771118+0000","last_active":"2024-03-26T21:51:38.069168+0000","last_peered":"2024-03-26T21:51:38.069168+0000","last_clean":"2024-03-26T21:51:38.069168+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:51:38.069168+0000","last_undegraded":"2024-03-26T21:51:38.069168+0000","last_fullsized":"2024-03-26T21:51:38.069168+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:29.770950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.726312+0000","last_clean_scrub_stamp":"2024-03-26T21:48:29.770950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997604+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":994,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023091+0000","last_change":"2024-03-26T21:48:15.882240+0000","last_active":"2024-03-26T21:51:39.023091+0000","last_peered":"2024-03-26T21:51:39.023091+0000","last_clean":"2024-03-26T21:51:39.023091+0000","last_became_active":"2024-03-26T21:36:51.717908+0000","last_became_peered":"2024-03-26T21:36:51.717908+0000","last_unstale":"2024-03-26T21:51:39.023091+0000","last_undegraded":"2024-03-26T21:51:39.023091+0000","last_fullsized":"2024-03-26T21:51:39.023091+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:15.881934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:15.881934+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":975,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379513+0000","last_change":"2024-03-26T21:49:31.653285+0000","last_active":"2024-03-26T21:51:44.379513+0000","last_peered":"2024-03-26T21:51:44.379513+0000","last_clean":"2024-03-26T21:51:44.379513+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:51:44.379513+0000","last_undegraded":"2024-03-26T21:51:44.379513+0000","last_fullsized":"2024-03-26T21:51:44.379513+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:31.653167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:49:31.653167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":771,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378949+0000","last_change":"2024-03-26T21:48:58.968424+0000","last_active":"2024-03-26T21:51:44.378949+0000","last_peered":"2024-03-26T21:51:44.378949+0000","last_clean":"2024-03-26T21:51:44.378949+0000","last_became_active":"2024-03-26T21:48:58.968010+0000","last_became_peered":"2024-03-26T21:48:58.968010+0000","last_unstale":"2024-03-26T21:51:44.378949+0000","last_undegraded":"2024-03-26T21:51:44.378949+0000","last_fullsized":"2024-03-26T21:51:44.378949+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:04.064025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:55.277361+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.064025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760703+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":934,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022840+0000","last_change":"2024-03-26T21:48:58.983049+0000","last_active":"2024-03-26T21:51:39.022840+0000","last_peered":"2024-03-26T21:51:39.022840+0000","last_clean":"2024-03-26T21:51:39.022840+0000","last_became_active":"2024-03-26T21:48:58.982744+0000","last_became_peered":"2024-03-26T21:48:58.982744+0000","last_unstale":"2024-03-26T21:51:39.022840+0000","last_undegraded":"2024-03-26T21:51:39.022840+0000","last_fullsized":"2024-03-26T21:51:39.022840+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.013772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.013772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1034,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.081762+0000","last_change":"2024-03-26T21:49:29.981307+0000","last_active":"2024-03-26T21:51:38.081762+0000","last_peered":"2024-03-26T21:51:38.081762+0000","last_clean":"2024-03-26T21:51:38.081762+0000","last_became_active":"2024-03-26T21:49:29.980914+0000","last_became_peered":"2024-03-26T21:49:29.980914+0000","last_unstale":"2024-03-26T21:51:38.081762+0000","last_undegraded":"2024-03-26T21:51:38.081762+0000","last_fullsized":"2024-03-26T21:51:38.081762+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'44","last_scrub_stamp":"2024-03-26T21:48:01.054345+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:48:01.054345+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":898,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068760+0000","last_change":"2024-03-26T21:48:44.612681+0000","last_active":"2024-03-26T21:51:38.068760+0000","last_peered":"2024-03-26T21:51:38.068760+0000","last_clean":"2024-03-26T21:51:38.068760+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:51:38.068760+0000","last_undegraded":"2024-03-26T21:51:38.068760+0000","last_fullsized":"2024-03-26T21:51:38.068760+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.612470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.228014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":85,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.021602+0000","last_change":"2024-03-26T21:48:58.986135+0000","last_active":"2024-03-26T21:51:39.021602+0000","last_peered":"2024-03-26T21:51:39.021602+0000","last_clean":"2024-03-26T21:51:39.021602+0000","last_became_active":"2024-03-26T21:48:58.985788+0000","last_became_peered":"2024-03-26T21:48:58.985788+0000","last_unstale":"2024-03-26T21:51:39.021602+0000","last_undegraded":"2024-03-26T21:51:39.021602+0000","last_fullsized":"2024-03-26T21:51:39.021602+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.055535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:01.366789+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.055535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":869,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065168+0000","last_change":"2024-03-26T21:48:58.984676+0000","last_active":"2024-03-26T21:51:38.065168+0000","last_peered":"2024-03-26T21:51:38.065168+0000","last_clean":"2024-03-26T21:51:38.065168+0000","last_became_active":"2024-03-26T21:48:58.983980+0000","last_became_peered":"2024-03-26T21:48:58.983980+0000","last_unstale":"2024-03-26T21:51:38.065168+0000","last_undegraded":"2024-03-26T21:51:38.065168+0000","last_fullsized":"2024-03-26T21:51:38.065168+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.896982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.896982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":85,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.021452+0000","last_change":"2024-03-26T21:48:58.977207+0000","last_active":"2024-03-26T21:51:39.021452+0000","last_peered":"2024-03-26T21:51:39.021452+0000","last_clean":"2024-03-26T21:51:39.021452+0000","last_became_active":"2024-03-26T21:48:58.976826+0000","last_became_peered":"2024-03-26T21:48:58.976826+0000","last_unstale":"2024-03-26T21:51:39.021452+0000","last_undegraded":"2024-03-26T21:51:39.021452+0000","last_fullsized":"2024-03-26T21:51:39.021452+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.129463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.129463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":708,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380216+0000","last_change":"2024-03-26T21:48:36.706426+0000","last_active":"2024-03-26T21:51:44.380216+0000","last_peered":"2024-03-26T21:51:44.380216+0000","last_clean":"2024-03-26T21:51:44.380216+0000","last_became_active":"2024-03-26T21:44:49.411976+0000","last_became_peered":"2024-03-26T21:44:49.411976+0000","last_unstale":"2024-03-26T21:51:44.380216+0000","last_undegraded":"2024-03-26T21:51:44.380216+0000","last_fullsized":"2024-03-26T21:51:44.380216+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'23","last_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:48:36.706278+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760762+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":928,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068073+0000","last_change":"2024-03-26T21:48:35.727970+0000","last_active":"2024-03-26T21:51:38.068073+0000","last_peered":"2024-03-26T21:51:38.068073+0000","last_clean":"2024-03-26T21:51:38.068073+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:51:38.068073+0000","last_undegraded":"2024-03-26T21:51:38.068073+0000","last_fullsized":"2024-03-26T21:51:38.068073+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:48:35.727841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.228047+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068517+0000","last_change":"2024-03-26T21:48:05.014565+0000","last_active":"2024-03-26T21:51:38.068517+0000","last_peered":"2024-03-26T21:51:38.068517+0000","last_clean":"2024-03-26T21:51:38.068517+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:51:38.068517+0000","last_undegraded":"2024-03-26T21:51:38.068517+0000","last_fullsized":"2024-03-26T21:51:38.068517+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:05.014452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:05.014452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997419+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"277'14","reported_seq":1161,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065434+0000","last_change":"2024-03-26T21:47:56.947809+0000","last_active":"2024-03-26T21:51:38.065434+0000","last_peered":"2024-03-26T21:51:38.065434+0000","last_clean":"2024-03-26T21:51:38.065434+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:51:38.065434+0000","last_undegraded":"2024-03-26T21:51:38.065434+0000","last_fullsized":"2024-03-26T21:51:38.065434+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:47:56.947620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:56.947620+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":1028,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.377672+0000","last_change":"2024-03-26T21:49:38.645618+0000","last_active":"2024-03-26T21:51:44.377672+0000","last_peered":"2024-03-26T21:51:44.377672+0000","last_clean":"2024-03-26T21:51:44.377672+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:51:44.377672+0000","last_undegraded":"2024-03-26T21:51:44.377672+0000","last_fullsized":"2024-03-26T21:51:44.377672+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:49:38.645456+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703319+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":937,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068546+0000","last_change":"2024-03-26T21:48:14.901387+0000","last_active":"2024-03-26T21:51:38.068546+0000","last_peered":"2024-03-26T21:51:38.068546+0000","last_clean":"2024-03-26T21:51:38.068546+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:51:38.068546+0000","last_undegraded":"2024-03-26T21:51:38.068546+0000","last_fullsized":"2024-03-26T21:51:38.068546+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:48:14.901301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.901301+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":943,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378816+0000","last_change":"2024-03-26T21:49:35.727593+0000","last_active":"2024-03-26T21:51:44.378816+0000","last_peered":"2024-03-26T21:51:44.378816+0000","last_clean":"2024-03-26T21:51:44.378816+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:51:44.378816+0000","last_undegraded":"2024-03-26T21:51:44.378816+0000","last_fullsized":"2024-03-26T21:51:44.378816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:35.727500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:49:35.727500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703282+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":320,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069451+0000","last_change":"2024-03-26T21:48:59.002050+0000","last_active":"2024-03-26T21:51:38.069451+0000","last_peered":"2024-03-26T21:51:38.069451+0000","last_clean":"2024-03-26T21:51:38.069451+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:51:38.069451+0000","last_undegraded":"2024-03-26T21:51:38.069451+0000","last_fullsized":"2024-03-26T21:51:38.069451+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'12","last_scrub_stamp":"2024-03-26T21:47:59.714477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.714477+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997489+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":72,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022279+0000","last_change":"2024-03-26T21:49:29.977391+0000","last_active":"2024-03-26T21:51:39.022279+0000","last_peered":"2024-03-26T21:51:39.022279+0000","last_clean":"2024-03-26T21:51:39.022279+0000","last_became_active":"2024-03-26T21:49:29.976836+0000","last_became_peered":"2024-03-26T21:49:29.976836+0000","last_unstale":"2024-03-26T21:51:39.022279+0000","last_undegraded":"2024-03-26T21:51:39.022279+0000","last_fullsized":"2024-03-26T21:51:39.022279+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.291965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":958,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378269+0000","last_change":"2024-03-26T21:49:41.705007+0000","last_active":"2024-03-26T21:51:44.378269+0000","last_peered":"2024-03-26T21:51:44.378269+0000","last_clean":"2024-03-26T21:51:44.378269+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:51:44.378269+0000","last_undegraded":"2024-03-26T21:51:44.378269+0000","last_fullsized":"2024-03-26T21:51:44.378269+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:41.704871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:49:41.704871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":873,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067178+0000","last_change":"2024-03-26T21:48:21.912464+0000","last_active":"2024-03-26T21:51:38.067178+0000","last_peered":"2024-03-26T21:51:38.067178+0000","last_clean":"2024-03-26T21:51:38.067178+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:51:38.067178+0000","last_undegraded":"2024-03-26T21:51:38.067178+0000","last_fullsized":"2024-03-26T21:51:38.067178+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:21.912248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:27.542602+0000","last_clean_scrub_stamp":"2024-03-26T21:48:21.912248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227961+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1011,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023243+0000","last_change":"2024-03-26T21:48:44.101005+0000","last_active":"2024-03-26T21:51:39.023243+0000","last_peered":"2024-03-26T21:51:39.023243+0000","last_clean":"2024-03-26T21:51:39.023243+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:51:39.023243+0000","last_undegraded":"2024-03-26T21:51:39.023243+0000","last_fullsized":"2024-03-26T21:51:39.023243+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:48:44.100867+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.100867+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":910,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065556+0000","last_change":"2024-03-26T21:48:03.921381+0000","last_active":"2024-03-26T21:51:38.065556+0000","last_peered":"2024-03-26T21:51:38.065556+0000","last_clean":"2024-03-26T21:51:38.065556+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:51:38.065556+0000","last_undegraded":"2024-03-26T21:51:38.065556+0000","last_fullsized":"2024-03-26T21:51:38.065556+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.921253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.921253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":289,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069605+0000","last_change":"2024-03-26T21:48:11.774249+0000","last_active":"2024-03-26T21:51:38.069605+0000","last_peered":"2024-03-26T21:51:38.069605+0000","last_clean":"2024-03-26T21:51:38.069605+0000","last_became_active":"2024-03-26T21:44:49.430580+0000","last_became_peered":"2024-03-26T21:44:49.430580+0000","last_unstale":"2024-03-26T21:51:38.069605+0000","last_undegraded":"2024-03-26T21:51:38.069605+0000","last_fullsized":"2024-03-26T21:51:38.069605+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:11.774125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:11.774125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":572,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379812+0000","last_change":"2024-03-26T21:48:58.984295+0000","last_active":"2024-03-26T21:51:44.379812+0000","last_peered":"2024-03-26T21:51:44.379812+0000","last_clean":"2024-03-26T21:51:44.379812+0000","last_became_active":"2024-03-26T21:48:58.983139+0000","last_became_peered":"2024-03-26T21:48:58.983139+0000","last_unstale":"2024-03-26T21:51:44.379812+0000","last_undegraded":"2024-03-26T21:51:44.379812+0000","last_fullsized":"2024-03-26T21:51:44.379812+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.903930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.903930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":1027,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378395+0000","last_change":"2024-03-26T21:50:00.738089+0000","last_active":"2024-03-26T21:51:44.378395+0000","last_peered":"2024-03-26T21:51:44.378395+0000","last_clean":"2024-03-26T21:51:44.378395+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:51:44.378395+0000","last_undegraded":"2024-03-26T21:51:44.378395+0000","last_fullsized":"2024-03-26T21:51:44.378395+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:50:00.737954+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:50:00.737954+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":949,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022737+0000","last_change":"2024-03-26T21:47:21.704189+0000","last_active":"2024-03-26T21:51:39.022737+0000","last_peered":"2024-03-26T21:51:39.022737+0000","last_clean":"2024-03-26T21:51:39.022737+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:51:39.022737+0000","last_undegraded":"2024-03-26T21:51:39.022737+0000","last_fullsized":"2024-03-26T21:51:39.022737+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:21.704073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:21.704073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"462'14","reported_seq":387,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.070334+0000","last_change":"2024-03-26T21:48:58.993774+0000","last_active":"2024-03-26T21:51:38.070334+0000","last_peered":"2024-03-26T21:51:38.070334+0000","last_clean":"2024-03-26T21:51:38.070334+0000","last_became_active":"2024-03-26T21:48:58.989325+0000","last_became_peered":"2024-03-26T21:48:58.989325+0000","last_unstale":"2024-03-26T21:51:38.070334+0000","last_undegraded":"2024-03-26T21:51:38.070334+0000","last_fullsized":"2024-03-26T21:51:38.070334+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'13","last_scrub_stamp":"2024-03-26T21:48:33.859117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.859117+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.025391+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":642,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379227+0000","last_change":"2024-03-26T21:48:58.970848+0000","last_active":"2024-03-26T21:51:44.379227+0000","last_peered":"2024-03-26T21:51:44.379227+0000","last_clean":"2024-03-26T21:51:44.379227+0000","last_became_active":"2024-03-26T21:48:58.970509+0000","last_became_peered":"2024-03-26T21:48:58.970509+0000","last_unstale":"2024-03-26T21:51:44.379227+0000","last_undegraded":"2024-03-26T21:51:44.379227+0000","last_fullsized":"2024-03-26T21:51:44.379227+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.705720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:49.010010+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.705720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760536+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":316,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.070059+0000","last_change":"2024-03-26T21:48:12.762664+0000","last_active":"2024-03-26T21:51:38.070059+0000","last_peered":"2024-03-26T21:51:38.070059+0000","last_clean":"2024-03-26T21:51:38.070059+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:51:38.070059+0000","last_undegraded":"2024-03-26T21:51:38.070059+0000","last_fullsized":"2024-03-26T21:51:38.070059+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'25","last_scrub_stamp":"2024-03-26T21:48:12.762526+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.762526+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997453+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":889,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066278+0000","last_change":"2024-03-26T21:48:58.987967+0000","last_active":"2024-03-26T21:51:38.066278+0000","last_peered":"2024-03-26T21:51:38.066278+0000","last_clean":"2024-03-26T21:51:38.066278+0000","last_became_active":"2024-03-26T21:48:58.987113+0000","last_became_peered":"2024-03-26T21:48:58.987113+0000","last_unstale":"2024-03-26T21:51:38.066278+0000","last_undegraded":"2024-03-26T21:51:38.066278+0000","last_fullsized":"2024-03-26T21:51:38.066278+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.931897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.931897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1234,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378155+0000","last_change":"2024-03-26T21:50:05.674011+0000","last_active":"2024-03-26T21:51:44.378155+0000","last_peered":"2024-03-26T21:51:44.378155+0000","last_clean":"2024-03-26T21:51:44.378155+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:51:44.378155+0000","last_undegraded":"2024-03-26T21:51:44.378155+0000","last_fullsized":"2024-03-26T21:51:44.378155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:50:05.673903+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:29:47.418949+0000","last_clean_scrub_stamp":"2024-03-26T21:50:05.673903+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":946,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022761+0000","last_change":"2024-03-26T21:47:23.698976+0000","last_active":"2024-03-26T21:51:39.022761+0000","last_peered":"2024-03-26T21:51:39.022761+0000","last_clean":"2024-03-26T21:51:39.022761+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:51:39.022761+0000","last_undegraded":"2024-03-26T21:51:39.022761+0000","last_fullsized":"2024-03-26T21:51:39.022761+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:47:23.698870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:47:23.698870+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":678,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379984+0000","last_change":"2024-03-26T21:47:22.503245+0000","last_active":"2024-03-26T21:51:44.379984+0000","last_peered":"2024-03-26T21:51:44.379984+0000","last_clean":"2024-03-26T21:51:44.379984+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:51:44.379984+0000","last_undegraded":"2024-03-26T21:51:44.379984+0000","last_fullsized":"2024-03-26T21:51:44.379984+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'24","last_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:47:22.503147+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760426+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":646,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378484+0000","last_change":"2024-03-26T21:48:39.735268+0000","last_active":"2024-03-26T21:51:44.378484+0000","last_peered":"2024-03-26T21:51:44.378484+0000","last_clean":"2024-03-26T21:51:44.378484+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:51:44.378484+0000","last_undegraded":"2024-03-26T21:51:44.378484+0000","last_fullsized":"2024-03-26T21:51:44.378484+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:48:39.735146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":970,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.081817+0000","last_change":"2024-03-26T21:48:24.700288+0000","last_active":"2024-03-26T21:51:38.081817+0000","last_peered":"2024-03-26T21:51:38.081817+0000","last_clean":"2024-03-26T21:51:38.081817+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:51:38.081817+0000","last_undegraded":"2024-03-26T21:51:38.081817+0000","last_fullsized":"2024-03-26T21:51:38.081817+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:48:24.700160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161481+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":289,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.077446+0000","last_change":"2024-03-26T21:47:18.309429+0000","last_active":"2024-03-26T21:51:38.077446+0000","last_peered":"2024-03-26T21:51:38.077446+0000","last_clean":"2024-03-26T21:51:38.077446+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:51:38.077446+0000","last_undegraded":"2024-03-26T21:51:38.077446+0000","last_fullsized":"2024-03-26T21:51:38.077446+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:18.309265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:47:18.309265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997257+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":1004,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.082020+0000","last_change":"2024-03-26T21:48:59.005271+0000","last_active":"2024-03-26T21:51:38.082020+0000","last_peered":"2024-03-26T21:51:38.082020+0000","last_clean":"2024-03-26T21:51:38.082020+0000","last_became_active":"2024-03-26T21:48:58.979497+0000","last_became_peered":"2024-03-26T21:48:58.979497+0000","last_unstale":"2024-03-26T21:51:38.082020+0000","last_undegraded":"2024-03-26T21:51:38.082020+0000","last_fullsized":"2024-03-26T21:51:38.082020+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:30.679017+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:48:30.679017+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":888,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067280+0000","last_change":"2024-03-26T21:48:13.873400+0000","last_active":"2024-03-26T21:51:38.067280+0000","last_peered":"2024-03-26T21:51:38.067280+0000","last_clean":"2024-03-26T21:51:38.067280+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:51:38.067280+0000","last_undegraded":"2024-03-26T21:51:38.067280+0000","last_fullsized":"2024-03-26T21:51:38.067280+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:13.873265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:13.873265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":289,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068281+0000","last_change":"2024-03-26T21:48:08.702648+0000","last_active":"2024-03-26T21:51:38.068281+0000","last_peered":"2024-03-26T21:51:38.068281+0000","last_clean":"2024-03-26T21:51:38.068281+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:51:38.068281+0000","last_undegraded":"2024-03-26T21:51:38.068281+0000","last_fullsized":"2024-03-26T21:51:38.068281+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:08.702212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:08.702212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997537+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":926,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.021850+0000","last_change":"2024-03-26T21:48:12.977290+0000","last_active":"2024-03-26T21:51:39.021850+0000","last_peered":"2024-03-26T21:51:39.021850+0000","last_clean":"2024-03-26T21:51:39.021850+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:51:39.021850+0000","last_undegraded":"2024-03-26T21:51:39.021850+0000","last_fullsized":"2024-03-26T21:51:39.021850+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.977168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.977168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":321,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.077582+0000","last_change":"2024-03-26T21:48:38.975837+0000","last_active":"2024-03-26T21:51:38.077582+0000","last_peered":"2024-03-26T21:51:38.077582+0000","last_clean":"2024-03-26T21:51:38.077582+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:51:38.077582+0000","last_undegraded":"2024-03-26T21:51:38.077582+0000","last_fullsized":"2024-03-26T21:51:38.077582+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:38.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:38.975710+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997724+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":963,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378429+0000","last_change":"2024-03-26T21:49:55.627083+0000","last_active":"2024-03-26T21:51:44.378429+0000","last_peered":"2024-03-26T21:51:44.378429+0000","last_clean":"2024-03-26T21:51:44.378429+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:51:44.378429+0000","last_undegraded":"2024-03-26T21:51:44.378429+0000","last_fullsized":"2024-03-26T21:51:44.378429+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:55.626947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:38:52.919189+0000","last_clean_scrub_stamp":"2024-03-26T21:49:55.626947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703510+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":865,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065586+0000","last_change":"2024-03-26T21:48:12.894397+0000","last_active":"2024-03-26T21:51:38.065586+0000","last_peered":"2024-03-26T21:51:38.065586+0000","last_clean":"2024-03-26T21:51:38.065586+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:51:38.065586+0000","last_undegraded":"2024-03-26T21:51:38.065586+0000","last_fullsized":"2024-03-26T21:51:38.065586+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:12.894257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.894257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":297,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068378+0000","last_change":"2024-03-26T21:48:37.104233+0000","last_active":"2024-03-26T21:51:38.068378+0000","last_peered":"2024-03-26T21:51:38.068378+0000","last_clean":"2024-03-26T21:51:38.068378+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:51:38.068378+0000","last_undegraded":"2024-03-26T21:51:38.068378+0000","last_fullsized":"2024-03-26T21:51:38.068378+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:37.104108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:48:37.104108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997670+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":309,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.067019+0000","last_change":"2024-03-26T21:48:42.913651+0000","last_active":"2024-03-26T21:51:38.067019+0000","last_peered":"2024-03-26T21:51:38.067019+0000","last_clean":"2024-03-26T21:51:38.067019+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:51:38.067019+0000","last_undegraded":"2024-03-26T21:51:38.067019+0000","last_fullsized":"2024-03-26T21:51:38.067019+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'8","last_scrub_stamp":"2024-03-26T21:48:42.913438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:42.913438+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997654+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":950,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023644+0000","last_change":"2024-03-26T21:48:10.939565+0000","last_active":"2024-03-26T21:51:39.023644+0000","last_peered":"2024-03-26T21:51:39.023644+0000","last_clean":"2024-03-26T21:51:39.023644+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:51:39.023644+0000","last_undegraded":"2024-03-26T21:51:39.023644+0000","last_fullsized":"2024-03-26T21:51:39.023644+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:10.939233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:10.939233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":563,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379687+0000","last_change":"2024-03-26T21:49:29.976926+0000","last_active":"2024-03-26T21:51:44.379687+0000","last_peered":"2024-03-26T21:51:44.379687+0000","last_clean":"2024-03-26T21:51:44.379687+0000","last_became_active":"2024-03-26T21:49:29.976388+0000","last_became_peered":"2024-03-26T21:49:29.976388+0000","last_unstale":"2024-03-26T21:51:44.379687+0000","last_undegraded":"2024-03-26T21:51:44.379687+0000","last_fullsized":"2024-03-26T21:51:44.379687+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.476531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":879,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065351+0000","last_change":"2024-03-26T21:48:58.980520+0000","last_active":"2024-03-26T21:51:38.065351+0000","last_peered":"2024-03-26T21:51:38.065351+0000","last_clean":"2024-03-26T21:51:38.065351+0000","last_became_active":"2024-03-26T21:48:58.980151+0000","last_became_peered":"2024-03-26T21:48:58.980151+0000","last_unstale":"2024-03-26T21:51:38.065351+0000","last_undegraded":"2024-03-26T21:51:38.065351+0000","last_fullsized":"2024-03-26T21:51:38.065351+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:22.860450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:22.860450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227872+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1003,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378939+0000","last_change":"2024-03-26T21:49:50.705427+0000","last_active":"2024-03-26T21:51:44.378939+0000","last_peered":"2024-03-26T21:51:44.378939+0000","last_clean":"2024-03-26T21:51:44.378939+0000","last_became_active":"2024-03-26T21:49:29.980233+0000","last_became_peered":"2024-03-26T21:49:29.980233+0000","last_unstale":"2024-03-26T21:51:44.378939+0000","last_undegraded":"2024-03-26T21:51:44.378939+0000","last_fullsized":"2024-03-26T21:51:44.378939+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:49:50.705277+0000","last_deep_scrub":"167'16","last_deep_scrub_stamp":"2024-03-26T21:34:28.014533+0000","last_clean_scrub_stamp":"2024-03-26T21:49:50.705277+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703480+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":289,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066824+0000","last_change":"2024-03-26T21:48:26.761549+0000","last_active":"2024-03-26T21:51:38.066824+0000","last_peered":"2024-03-26T21:51:38.066824+0000","last_clean":"2024-03-26T21:51:38.066824+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:51:38.066824+0000","last_undegraded":"2024-03-26T21:51:38.066824+0000","last_fullsized":"2024-03-26T21:51:38.066824+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:26.760812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:26.760812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997619+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":947,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023404+0000","last_change":"2024-03-26T21:48:58.986355+0000","last_active":"2024-03-26T21:51:39.023404+0000","last_peered":"2024-03-26T21:51:39.023404+0000","last_clean":"2024-03-26T21:51:39.023404+0000","last_became_active":"2024-03-26T21:48:58.985913+0000","last_became_peered":"2024-03-26T21:48:58.985913+0000","last_unstale":"2024-03-26T21:51:39.023404+0000","last_undegraded":"2024-03-26T21:51:39.023404+0000","last_fullsized":"2024-03-26T21:51:39.023404+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:48:45.073486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":963,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.096122+0000","last_change":"2024-03-26T21:48:23.018053+0000","last_active":"2024-03-26T21:51:38.096122+0000","last_peered":"2024-03-26T21:51:38.096122+0000","last_clean":"2024-03-26T21:51:38.096122+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:51:38.096122+0000","last_undegraded":"2024-03-26T21:51:38.096122+0000","last_fullsized":"2024-03-26T21:51:38.096122+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:23.017941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:23.017941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161496+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1034,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.023119+0000","last_change":"2024-03-26T21:48:09.994077+0000","last_active":"2024-03-26T21:51:39.023119+0000","last_peered":"2024-03-26T21:51:39.023119+0000","last_clean":"2024-03-26T21:51:39.023119+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:51:39.023119+0000","last_undegraded":"2024-03-26T21:51:39.023119+0000","last_fullsized":"2024-03-26T21:51:39.023119+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'20","last_scrub_stamp":"2024-03-26T21:48:09.993945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.993945+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":997,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.075938+0000","last_change":"2024-03-26T21:48:06.715372+0000","last_active":"2024-03-26T21:51:38.075938+0000","last_peered":"2024-03-26T21:51:38.075938+0000","last_clean":"2024-03-26T21:51:38.075938+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:51:38.075938+0000","last_undegraded":"2024-03-26T21:51:38.075938+0000","last_fullsized":"2024-03-26T21:51:38.075938+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:48:06.715234+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:48:06.715234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"468'15","reported_seq":547,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378968+0000","last_change":"2024-03-26T21:49:29.976278+0000","last_active":"2024-03-26T21:51:44.378968+0000","last_peered":"2024-03-26T21:51:44.378968+0000","last_clean":"2024-03-26T21:51:44.378968+0000","last_became_active":"2024-03-26T21:49:29.975921+0000","last_became_peered":"2024-03-26T21:49:29.975921+0000","last_unstale":"2024-03-26T21:51:44.378968+0000","last_undegraded":"2024-03-26T21:51:44.378968+0000","last_fullsized":"2024-03-26T21:51:44.378968+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'14","last_scrub_stamp":"2024-03-26T21:48:32.327875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:32.327875+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760801+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":879,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.066314+0000","last_change":"2024-03-26T21:48:58.998136+0000","last_active":"2024-03-26T21:51:38.066314+0000","last_peered":"2024-03-26T21:51:38.066314+0000","last_clean":"2024-03-26T21:51:38.066314+0000","last_became_active":"2024-03-26T21:48:58.991425+0000","last_became_peered":"2024-03-26T21:48:58.991425+0000","last_unstale":"2024-03-26T21:51:38.066314+0000","last_undegraded":"2024-03-26T21:51:38.066314+0000","last_fullsized":"2024-03-26T21:51:38.066314+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:48:04.880983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:04.880983+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":315,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068744+0000","last_change":"2024-03-26T21:47:15.327808+0000","last_active":"2024-03-26T21:51:38.068744+0000","last_peered":"2024-03-26T21:51:38.068744+0000","last_clean":"2024-03-26T21:51:38.068744+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:51:38.068744+0000","last_undegraded":"2024-03-26T21:51:38.068744+0000","last_fullsized":"2024-03-26T21:51:38.068744+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'23","last_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_deep_scrub":"421'23","last_deep_scrub_stamp":"2024-03-26T21:47:15.327628+0000","last_clean_scrub_stamp":"2024-03-26T21:47:15.327628+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997200+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":642,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379357+0000","last_change":"2024-03-26T21:48:58.973845+0000","last_active":"2024-03-26T21:51:44.379357+0000","last_peered":"2024-03-26T21:51:44.379357+0000","last_clean":"2024-03-26T21:51:44.379357+0000","last_became_active":"2024-03-26T21:48:58.973428+0000","last_became_peered":"2024-03-26T21:48:58.973428+0000","last_unstale":"2024-03-26T21:51:44.379357+0000","last_undegraded":"2024-03-26T21:51:44.379357+0000","last_fullsized":"2024-03-26T21:51:44.379357+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:07.663530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:48:07.663530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760630+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":666,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378703+0000","last_change":"2024-03-26T21:48:58.983537+0000","last_active":"2024-03-26T21:51:44.378703+0000","last_peered":"2024-03-26T21:51:44.378703+0000","last_clean":"2024-03-26T21:51:44.378703+0000","last_became_active":"2024-03-26T21:48:58.982850+0000","last_became_peered":"2024-03-26T21:48:58.982850+0000","last_unstale":"2024-03-26T21:51:44.378703+0000","last_undegraded":"2024-03-26T21:51:44.378703+0000","last_fullsized":"2024-03-26T21:51:44.378703+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:16.766699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:48:16.766699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":290,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069683+0000","last_change":"2024-03-26T21:48:20.092767+0000","last_active":"2024-03-26T21:51:38.069683+0000","last_peered":"2024-03-26T21:51:38.069683+0000","last_clean":"2024-03-26T21:51:38.069683+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:51:38.069683+0000","last_undegraded":"2024-03-26T21:51:38.069683+0000","last_fullsized":"2024-03-26T21:51:38.069683+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:20.092636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:20.092636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997505+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1077,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.081787+0000","last_change":"2024-03-26T21:48:59.014048+0000","last_active":"2024-03-26T21:51:38.081787+0000","last_peered":"2024-03-26T21:51:38.081787+0000","last_clean":"2024-03-26T21:51:38.081787+0000","last_became_active":"2024-03-26T21:48:58.977341+0000","last_became_peered":"2024-03-26T21:48:58.977341+0000","last_unstale":"2024-03-26T21:51:38.081787+0000","last_undegraded":"2024-03-26T21:51:38.081787+0000","last_fullsized":"2024-03-26T21:51:38.081787+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:09.176289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:09.176289+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161374+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":960,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022806+0000","last_change":"2024-03-26T21:48:40.024866+0000","last_active":"2024-03-26T21:51:39.022806+0000","last_peered":"2024-03-26T21:51:39.022806+0000","last_clean":"2024-03-26T21:51:39.022806+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:51:39.022806+0000","last_undegraded":"2024-03-26T21:51:39.022806+0000","last_fullsized":"2024-03-26T21:51:39.022806+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:40.024681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:48:40.024681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":959,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.097162+0000","last_change":"2024-03-26T21:48:02.668719+0000","last_active":"2024-03-26T21:51:38.097162+0000","last_peered":"2024-03-26T21:51:38.097162+0000","last_clean":"2024-03-26T21:51:38.097162+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:51:38.097162+0000","last_undegraded":"2024-03-26T21:51:38.097162+0000","last_fullsized":"2024-03-26T21:51:38.097162+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:02.668586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:48:02.668586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161465+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":638,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380022+0000","last_change":"2024-03-26T21:48:14.764275+0000","last_active":"2024-03-26T21:51:44.380022+0000","last_peered":"2024-03-26T21:51:44.380022+0000","last_clean":"2024-03-26T21:51:44.380022+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:51:44.380022+0000","last_undegraded":"2024-03-26T21:51:44.380022+0000","last_fullsized":"2024-03-26T21:51:44.380022+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:14.764160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:14.764160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760779+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":672,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.380117+0000","last_change":"2024-03-26T21:47:59.581024+0000","last_active":"2024-03-26T21:51:44.380117+0000","last_peered":"2024-03-26T21:51:44.380117+0000","last_clean":"2024-03-26T21:51:44.380117+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:51:44.380117+0000","last_undegraded":"2024-03-26T21:51:44.380117+0000","last_fullsized":"2024-03-26T21:51:44.380117+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'7","last_scrub_stamp":"2024-03-26T21:47:59.580889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:47:59.580889+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":975,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379411+0000","last_change":"2024-03-26T21:50:09.758825+0000","last_active":"2024-03-26T21:51:44.379411+0000","last_peered":"2024-03-26T21:51:44.379411+0000","last_clean":"2024-03-26T21:51:44.379411+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:51:44.379411+0000","last_undegraded":"2024-03-26T21:51:44.379411+0000","last_fullsized":"2024-03-26T21:51:44.379411+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:09.758697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:50:09.758697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":605,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.065614+0000","last_change":"2024-03-26T21:47:58.927137+0000","last_active":"2024-03-26T21:51:38.065614+0000","last_peered":"2024-03-26T21:51:38.065614+0000","last_clean":"2024-03-26T21:51:38.065614+0000","last_became_active":"2024-03-26T21:41:25.658125+0000","last_became_peered":"2024-03-26T21:41:25.658125+0000","last_unstale":"2024-03-26T21:51:38.065614+0000","last_undegraded":"2024-03-26T21:51:38.065614+0000","last_fullsized":"2024-03-26T21:51:38.065614+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.927008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.927008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:41.227758+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":962,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:46.750936+0000","last_change":"2024-03-26T21:51:46.750961+0000","last_active":"2024-03-26T21:51:46.750936+0000","last_peered":"2024-03-26T21:51:46.750936+0000","last_clean":"2024-03-26T21:51:46.750936+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:51:46.750936+0000","last_undegraded":"2024-03-26T21:51:46.750936+0000","last_fullsized":"2024-03-26T21:51:46.750936+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:51:46.750852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:51:46.750852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:10.415237+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":704,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379687+0000","last_change":"2024-03-26T21:48:27.673802+0000","last_active":"2024-03-26T21:51:44.379687+0000","last_peered":"2024-03-26T21:51:44.379687+0000","last_clean":"2024-03-26T21:51:44.379687+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:51:44.379687+0000","last_undegraded":"2024-03-26T21:51:44.379687+0000","last_fullsized":"2024-03-26T21:51:44.379687+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'19","last_scrub_stamp":"2024-03-26T21:48:27.673516+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:48:27.673516+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":975,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378001+0000","last_change":"2024-03-26T21:49:48.724032+0000","last_active":"2024-03-26T21:51:44.378001+0000","last_peered":"2024-03-26T21:51:44.378001+0000","last_clean":"2024-03-26T21:51:44.378001+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:51:44.378001+0000","last_undegraded":"2024-03-26T21:51:44.378001+0000","last_fullsized":"2024-03-26T21:51:44.378001+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:49:48.723773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:49:48.723773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703410+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":955,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.377660+0000","last_change":"2024-03-26T21:50:06.681825+0000","last_active":"2024-03-26T21:51:44.377660+0000","last_peered":"2024-03-26T21:51:44.377660+0000","last_clean":"2024-03-26T21:51:44.377660+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:51:44.377660+0000","last_undegraded":"2024-03-26T21:51:44.377660+0000","last_fullsized":"2024-03-26T21:51:44.377660+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_clean_scrub_stamp":"2024-03-26T21:50:06.681696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:44.703540+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":927,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022133+0000","last_change":"2024-03-26T21:48:03.977558+0000","last_active":"2024-03-26T21:51:39.022133+0000","last_peered":"2024-03-26T21:51:39.022133+0000","last_clean":"2024-03-26T21:51:39.022133+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:51:39.022133+0000","last_undegraded":"2024-03-26T21:51:39.022133+0000","last_fullsized":"2024-03-26T21:51:39.022133+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:03.977428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:03.977428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":345,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.077482+0000","last_change":"2024-03-26T21:48:59.006965+0000","last_active":"2024-03-26T21:51:38.077482+0000","last_peered":"2024-03-26T21:51:38.077482+0000","last_clean":"2024-03-26T21:51:38.077482+0000","last_became_active":"2024-03-26T21:48:58.977997+0000","last_became_peered":"2024-03-26T21:48:58.977997+0000","last_unstale":"2024-03-26T21:51:38.077482+0000","last_undegraded":"2024-03-26T21:51:38.077482+0000","last_fullsized":"2024-03-26T21:51:38.077482+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'27","last_scrub_stamp":"2024-03-26T21:48:34.910272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:34.910272+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997687+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":296,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.069086+0000","last_change":"2024-03-26T21:48:58.976105+0000","last_active":"2024-03-26T21:51:38.069086+0000","last_peered":"2024-03-26T21:51:38.069086+0000","last_clean":"2024-03-26T21:51:38.069086+0000","last_became_active":"2024-03-26T21:48:58.975714+0000","last_became_peered":"2024-03-26T21:48:58.975714+0000","last_unstale":"2024-03-26T21:51:38.069086+0000","last_undegraded":"2024-03-26T21:51:38.069086+0000","last_fullsized":"2024-03-26T21:51:38.069086+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:58.673609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:47:58.673609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":637,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.377964+0000","last_change":"2024-03-26T21:47:20.511937+0000","last_active":"2024-03-26T21:51:44.377964+0000","last_peered":"2024-03-26T21:51:44.377964+0000","last_clean":"2024-03-26T21:51:44.377964+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:51:44.377964+0000","last_undegraded":"2024-03-26T21:51:44.377964+0000","last_fullsized":"2024-03-26T21:51:44.377964+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:20.511844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:38.083526+0000","last_clean_scrub_stamp":"2024-03-26T21:47:20.511844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":289,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.068182+0000","last_change":"2024-03-26T21:47:25.356191+0000","last_active":"2024-03-26T21:51:38.068182+0000","last_peered":"2024-03-26T21:51:38.068182+0000","last_clean":"2024-03-26T21:51:38.068182+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:51:38.068182+0000","last_undegraded":"2024-03-26T21:51:38.068182+0000","last_fullsized":"2024-03-26T21:51:38.068182+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:25.356071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:47:25.356071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:38.997330+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":681,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.378814+0000","last_change":"2024-03-26T21:48:12.694117+0000","last_active":"2024-03-26T21:51:44.378814+0000","last_peered":"2024-03-26T21:51:44.378814+0000","last_clean":"2024-03-26T21:51:44.378814+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:51:44.378814+0000","last_undegraded":"2024-03-26T21:51:44.378814+0000","last_fullsized":"2024-03-26T21:51:44.378814+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'15","last_scrub_stamp":"2024-03-26T21:48:12.694019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:48:12.694019+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":946,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:39.022845+0000","last_change":"2024-03-26T21:48:58.978607+0000","last_active":"2024-03-26T21:51:39.022845+0000","last_peered":"2024-03-26T21:51:39.022845+0000","last_clean":"2024-03-26T21:51:39.022845+0000","last_became_active":"2024-03-26T21:48:58.978253+0000","last_became_peered":"2024-03-26T21:48:58.978253+0000","last_unstale":"2024-03-26T21:51:39.022845+0000","last_undegraded":"2024-03-26T21:51:39.022845+0000","last_fullsized":"2024-03-26T21:51:39.022845+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:33.984501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:31.663239+0000","last_clean_scrub_stamp":"2024-03-26T21:48:33.984501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":947,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-03-26T21:51:38.075702+0000","last_change":"2024-03-26T21:47:12.468409+0000","last_active":"2024-03-26T21:51:38.075702+0000","last_peered":"2024-03-26T21:51:38.075702+0000","last_clean":"2024-03-26T21:51:38.075702+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:51:38.075702+0000","last_undegraded":"2024-03-26T21:51:38.075702+0000","last_fullsized":"2024-03-26T21:51:38.075702+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:47:12.468179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:47:12.468179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:42.161290+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":644,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-26T21:51:44.379457+0000","last_change":"2024-03-26T21:48:44.829403+0000","last_active":"2024-03-26T21:51:44.379457+0000","last_peered":"2024-03-26T21:51:44.379457+0000","last_clean":"2024-03-26T21:51:44.379457+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:51:44.379457+0000","last_undegraded":"2024-03-26T21:51:44.379457+0000","last_fullsized":"2024-03-26T21:51:44.379457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:48:44.829181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:48:44.829181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:51:46.760874+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1267,"num_read_kb":1262,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":7130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1072,"ondisk_log_size":1072,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973821,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596344,"kb_used_data":1388,"kb_used_omap":6,"kb_used_meta":594937,"kb_avail":93148808,"statfs":{"total":95995035648,"available":95384379392,"internally_reserved":0,"allocated":1421312,"data_stored":1008093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":609215784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365089,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732280,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93012872,"statfs":{"total":95995035648,"available":95245180928,"internally_reserved":0,"allocated":1421312,"data_stored":1012673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575162,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974660,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92770492,"statfs":{"total":95995035648,"available":94996983808,"internally_reserved":0,"allocated":843776,"data_stored":417066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738687,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939140,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806012,"statfs":{"total":95995035648,"available":95033356288,"internally_reserved":0,"allocated":1433600,"data_stored":1007729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411637,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149998,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":984384,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":92760768,"statfs":{"total":95995035648,"available":94987026432,"internally_reserved":0,"allocated":839680,"data_stored":416587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1007155655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650595,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813188,"kb_used_data":824,"kb_used_omap":4,"kb_used_meta":812347,"kb_avail":92931964,"statfs":{"total":95995035648,"available":95162331136,"internally_reserved":0,"allocated":843776,"data_stored":423789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":831844003},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659428,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735864,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009288,"statfs":{"total":95995035648,"available":95241510912,"internally_reserved":0,"allocated":831488,"data_stored":422631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:51:49.334 INFO:tasks.thrashosds.thrasher:Setting 5.1a to [0, 1, 3] 2024-03-26T21:51:49.334 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1a', '0', '1', '3'] 2024-03-26T21:51:49.334 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 5.1a 0 1 3 2024-03-26T21:51:49.340 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:49.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:49.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:49.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:51:49.462 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:51:49.463 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:51:49.464 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:49 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:51:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: pgmap v1096: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:49.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:51:49.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:49 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:51:50.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 5.9 scrub starts 2024-03-26T21:51:50.441 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 5.9 scrub ok 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 5.9 scrub starts 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 5.9 scrub ok 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:51:50.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:50 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 5.9 scrub starts 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 5.9 scrub ok 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:51:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:50 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:51:50.504 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:50.507 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:51.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[40587]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:51.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:51:51.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:51:51.407 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[29406]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:51.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:51:51.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:51 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:51:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:51 smithi067 ceph-mon[33346]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:51 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:51:51.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:51 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:51:52.193 INFO:teuthology.orchestra.run.smithi055.stderr:noscrub is unset 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:51:52.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:52.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:51:52.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:52 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:51:52.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:52 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:51:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:52 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:51:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T21:51:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:52 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:51:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:52 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:51:53.247 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:51:53.247 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: osdmap e500: 8 total, 8 up, 7 in 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: 5.7 scrub starts 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: 5.7 scrub ok 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:51:53.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:53 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: osdmap e500: 8 total, 8 up, 7 in 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: 5.7 scrub starts 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: 5.7 scrub ok 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:51:53.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2921092892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: osdmap e500: 8 total, 8 up, 7 in 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: 5.7 scrub starts 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: 5.7 scrub ok 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:51:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:53 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:51:53.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T21:51:54.427 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 5.18 scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[40587]: 5.18 scrub ok 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 5.18 scrub starts 2024-03-26T21:51:54.428 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:54 smithi055 ceph-mon[29406]: 5.18 scrub ok 2024-03-26T21:51:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:51:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:51:54.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:51:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:51:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 5.18 scrub starts 2024-03-26T21:51:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:54 smithi067 ceph-mon[33346]: 5.18 scrub ok 2024-03-26T21:51:55.125 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T21:51:55.362 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:55 smithi055 ceph-mon[29406]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:55.362 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:55 smithi055 ceph-mon[40587]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:55.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:55 smithi067 ceph-mon[33346]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:51:55.566 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:55.601 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:51:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:set 5.1a pg_upmap mapping to [0,1,3] 2024-03-26T21:51:56.300 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[40587]: 4.1 scrub starts 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[40587]: 4.1 scrub ok 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]: dispatch 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[29406]: 4.1 scrub starts 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[29406]: 4.1 scrub ok 2024-03-26T21:51:56.301 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]: dispatch 2024-03-26T21:51:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:56 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:51:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:56 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:51:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:56 smithi067 ceph-mon[33346]: 4.1 scrub starts 2024-03-26T21:51:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:56 smithi067 ceph-mon[33346]: 4.1 scrub ok 2024-03-26T21:51:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]: dispatch 2024-03-26T21:51:57.390 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]': finished 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: osdmap e501: 8 total, 8 up, 7 in 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]': finished 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: osdmap e501: 8 total, 8 up, 7 in 2024-03-26T21:51:57.391 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:51:57.392 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:51:57.392 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:51:57.392 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:57 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2829801583' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1a", "id": [0, 1, 3]}]': finished 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: osdmap e501: 8 total, 8 up, 7 in 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:51:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:51:58.227 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:51:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: osdmap e502: 8 total, 8 up, 7 in 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:51:58.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: osdmap e502: 8 total, 8 up, 7 in 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:51:58.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: osdmap e502: 8 total, 8 up, 7 in 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:51:58.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:58 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:51:59.144 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:51:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:51:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:51:59.422 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[29406]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:59.422 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:59.422 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:51:59.422 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:51:59.422 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[40587]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:59.423 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:59.423 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:51:59.423 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:51:59 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:51:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:59 smithi067 ceph-mon[33346]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:51:59.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:51:59.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:59 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:51:59.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:51:59 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:51:59.835 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:51:59.835 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:00.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:52:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:52:00.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:00 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:52:00.536 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:52:00.537 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:00 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:52:00.564 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T21:52:00.760 INFO:teuthology.orchestra.run.smithi055.stderr:nodeep-scrub is unset 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: 4.1c scrub starts 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: 4.1c scrub ok 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:52:01.441 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: 4.1c scrub starts 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: 4.1c scrub ok 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:52:01.442 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:01 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:52:01.479 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:01.489 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: 4.1c scrub starts 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: 4.1c scrub ok 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/118403920' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: osdmap e503: 8 total, 8 up, 7 in 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:52:01.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:01 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:52:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:02 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:52:02.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:02 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:52:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:02 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:52:02.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:02 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:52:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:02 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:52:02.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:02 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:52:02.954 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T21:52:02.955 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-03-26T21:52:02.955 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd in 2 2024-03-26T21:52:03.354 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:03.355 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 5.b deep-scrub starts 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 5.b deep-scrub ok 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 6.17 deep-scrub starts 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 6.17 deep-scrub ok 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:52:03.355 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 5.b deep-scrub starts 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 5.b deep-scrub ok 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 6.17 deep-scrub starts 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 6.17 deep-scrub ok 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:03.356 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:03.739 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 5.b deep-scrub starts 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 5.b deep-scrub ok 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 6.17 deep-scrub starts 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 6.17 deep-scrub ok 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:03.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:04.056 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:04.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T21:52:04.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[29406]: 3.11 deep-scrub starts 2024-03-26T21:52:04.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[29406]: 3.11 deep-scrub ok 2024-03-26T21:52:04.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:52:04.306 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:52:04.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[40587]: 3.11 deep-scrub starts 2024-03-26T21:52:04.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[40587]: 3.11 deep-scrub ok 2024-03-26T21:52:04.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:52:04.307 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:04 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:52:04.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:04 smithi067 ceph-mon[33346]: 3.11 deep-scrub starts 2024-03-26T21:52:04.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:04 smithi067 ceph-mon[33346]: 3.11 deep-scrub ok 2024-03-26T21:52:04.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:04 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:52:04.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:04 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:52:04.798 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:05.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[40587]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:05.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[40587]: 4.1f deep-scrub starts 2024-03-26T21:52:05.466 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[40587]: 4.1f deep-scrub ok 2024-03-26T21:52:05.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[29406]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:05.466 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[29406]: 4.1f deep-scrub starts 2024-03-26T21:52:05.467 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:05 smithi055 ceph-mon[29406]: 4.1f deep-scrub ok 2024-03-26T21:52:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:05 smithi067 ceph-mon[33346]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:05 smithi067 ceph-mon[33346]: 4.1f deep-scrub starts 2024-03-26T21:52:05.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:05 smithi067 ceph-mon[33346]: 4.1f deep-scrub ok 2024-03-26T21:52:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:06 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:52:06.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:06 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:52:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:06 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:52:06.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:06 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:52:06.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:06 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:52:06.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:06 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:52:06.918 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:06 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:52:06.716+0000 7fb1ff446700 -1 mon.a@0(leader).osd e503 definitely_dead 0 2024-03-26T21:52:07.276 INFO:teuthology.orchestra.run.smithi055.stderr:marked in osd.2. 2024-03-26T21:52:07.488 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:07.489 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[40587]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[40587]: 3.16 scrub starts 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[40587]: 3.16 scrub ok 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[29406]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:07.489 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[29406]: 3.16 scrub starts 2024-03-26T21:52:07.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[29406]: 3.16 scrub ok 2024-03-26T21:52:07.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-26T21:52:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:07 smithi067 ceph-mon[33346]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:52:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:07 smithi067 ceph-mon[33346]: 3.16 scrub starts 2024-03-26T21:52:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:07 smithi067 ceph-mon[33346]: 3.16 scrub ok 2024-03-26T21:52:07.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-26T21:52:08.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: 6.b scrub starts 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[40587]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:52:08.512 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:52:08.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:52:08.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: 6.b scrub starts 2024-03-26T21:52:08.513 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-mon[29406]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:52:08.539 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1787812394' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: 6.b scrub starts 2024-03-26T21:52:08.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:08 smithi067 ceph-mon[33346]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T21:52:08.981 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-03-26T21:52:08.981 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-26T21:52:09.020 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:09.285 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:09.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: pgmap v1111: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:09.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: 6.b scrub ok 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: pgmap v1111: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: 6.b scrub ok 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:52:09.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:09 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: pgmap v1111: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: 6.b scrub ok 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:52:09.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:09 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:10.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:52:10.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:10 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:52:10.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:10 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:52:10.754 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:10.755 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:11.412 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T21:52:11.446 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 6.f deep-scrub starts 2024-03-26T21:52:11.565 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 6.f deep-scrub ok 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: pgmap v1113: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 59 B/s, 1 objects/s recovering 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 5.1 deep-scrub starts 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[40587]: 5.1 deep-scrub ok 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 6.f deep-scrub starts 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 6.f deep-scrub ok 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: pgmap v1113: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 59 B/s, 1 objects/s recovering 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:52:11.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:52:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 5.1 deep-scrub starts 2024-03-26T21:52:11.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:11 smithi055 ceph-mon[29406]: 5.1 deep-scrub ok 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 6.f deep-scrub starts 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 6.f deep-scrub ok 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: pgmap v1113: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 59 B/s, 1 objects/s recovering 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 5.1 deep-scrub starts 2024-03-26T21:52:11.641 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:11 smithi067 ceph-mon[33346]: 5.1 deep-scrub ok 2024-03-26T21:52:12.177 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:13.058 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: pgmap v1115: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 98 B/s, 3 objects/s recovering 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: pgmap v1115: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 98 B/s, 3 objects/s recovering 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:52:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:12 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: pgmap v1115: 132 pgs: 9 peering, 123 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 98 B/s, 3 objects/s recovering 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:52:13.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:12 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:52:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[40587]: 5.a scrub starts 2024-03-26T21:52:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[40587]: 5.a scrub ok 2024-03-26T21:52:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:52:14.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:52:14.066 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[29406]: 5.a scrub starts 2024-03-26T21:52:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[29406]: 5.a scrub ok 2024-03-26T21:52:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:52:14.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:13 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:52:14.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:13 smithi067 ceph-mon[33346]: 5.a scrub starts 2024-03-26T21:52:14.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:13 smithi067 ceph-mon[33346]: 5.a scrub ok 2024-03-26T21:52:14.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:13 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:52:14.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:13 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:52:14.420 INFO:teuthology.orchestra.run.smithi055.stdout:mon.a: {} 2024-03-26T21:52:14.430 INFO:teuthology.orchestra.run.smithi055.stdout:mon.b: {} 2024-03-26T21:52:14.441 INFO:teuthology.orchestra.run.smithi055.stdout:mon.c: {} 2024-03-26T21:52:14.442 INFO:teuthology.orchestra.run.smithi055.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:52:14.442 INFO:teuthology.orchestra.run.smithi055.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:52:14.442 INFO:teuthology.orchestra.run.smithi055.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T21:52:15.064 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:15.064 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194 B/s, 7 objects/s recovering 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:52:15.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:52:15.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:52:15.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1931659603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194 B/s, 7 objects/s recovering 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:52:15.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1931659603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194 B/s, 7 objects/s recovering 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:52:15.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:52:15.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:52:15.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:52:15.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:52:15.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1931659603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T21:52:15.807 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T21:52:16.066 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 2.1 scrub starts 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 2.1 scrub ok 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:52:16.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 2.1 scrub starts 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 2.1 scrub ok 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:52:16.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:15 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 2.1 scrub starts 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 2.1 scrub ok 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:52:16.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:15 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:52:16.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T21:52:16.293 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 155 B/s, 5 objects/s recovering 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]': finished 2024-03-26T21:52:17.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:16 smithi067 ceph-mon[33346]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 155 B/s, 5 objects/s recovering 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]': finished 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[29406]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 155 B/s, 5 objects/s recovering 2024-03-26T21:52:17.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]: dispatch 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-03-26T21:52:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 5]}]': finished 2024-03-26T21:52:17.319 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:16 smithi055 ceph-mon[40587]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T21:52:17.915 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:17.954 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:17.954 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:18.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:52:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:52:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:52:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:52:18.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:52:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:18 smithi067 ceph-mon[33346]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[29406]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:52:18.295 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:52:18.296 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-mon[40587]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T21:52:19.267 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: pgmap v1121: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:52:19.268 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: pgmap v1121: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:52:19.269 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:19 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:52:19.294 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: pgmap v1121: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:52:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:19 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:52:20.247 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 6.1f scrub starts 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:52:20.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 6.1f scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:52:20.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:20 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 6.1f scrub starts 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:52:20.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:52:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:52:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:52:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:20 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:52:21.024 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 6.1f scrub ok 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: pgmap v1123: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 3.8 scrub starts 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 3.8 scrub ok 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:52:21.298 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 6.1f scrub ok 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: pgmap v1123: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 3.8 scrub starts 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 3.8 scrub ok 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:52:21.299 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:21 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 6.1f scrub ok 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: pgmap v1123: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 3.8 scrub starts 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 3.8 scrub ok 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:52:21.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:21 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:52:22.022 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-26T21:52:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: 5.10 scrub starts 2024-03-26T21:52:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: 5.10 scrub ok 2024-03-26T21:52:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:52:22.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: 5.10 scrub starts 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: 5.10 scrub ok 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:52:22.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:22 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:52:22.472 INFO:teuthology.orchestra.run.smithi055.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:22.472 INFO:teuthology.orchestra.run.smithi055.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: 5.10 scrub starts 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: 5.10 scrub ok 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2046558425' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:22.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:52:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:22 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:52:23.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T21:52:23.126 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:23.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:23 smithi055 ceph-mon[40587]: pgmap v1125: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 703 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T21:52:23.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:23 smithi055 ceph-mon[40587]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:52:23.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:23 smithi055 ceph-mon[29406]: pgmap v1125: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 703 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T21:52:23.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:23 smithi055 ceph-mon[29406]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:52:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:23.464 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:23 smithi067 ceph-mon[33346]: pgmap v1125: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 703 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T21:52:23.464 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:23 smithi067 ceph-mon[33346]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T21:52:24.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 6.15 deep-scrub starts 2024-03-26T21:52:24.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 6.15 deep-scrub ok 2024-03-26T21:52:24.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 5.13 scrub starts 2024-03-26T21:52:24.401 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 5.13 scrub ok 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 6.15 deep-scrub starts 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 6.15 deep-scrub ok 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 5.13 scrub starts 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 5.13 scrub ok 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:52:24.402 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:24 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 6.15 deep-scrub starts 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 6.15 deep-scrub ok 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 5.13 scrub starts 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 5.13 scrub ok 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:52:24.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:24 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:52:25.169 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:25 smithi067 ceph-mon[33346]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 73 B/s, 3 objects/s recovering 2024-03-26T21:52:25.169 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:25 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:52:25.169 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:25 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:52:25.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[29406]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 73 B/s, 3 objects/s recovering 2024-03-26T21:52:25.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:52:25.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:52:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[40587]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 73 B/s, 3 objects/s recovering 2024-03-26T21:52:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T21:52:25.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:25 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:52:25.819 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:25.819 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:26.084 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-26T21:52:26.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:52:26.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:52:26.416 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:52:26.417 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:26 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:52:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:26 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:52:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:26 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:52:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:26 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T21:52:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:26 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:52:26.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:26 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:52:26.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T21:52:26.808 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:26.833 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T21:52:27.088 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:27.202 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.e deep-scrub starts 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.e deep-scrub ok 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 56 B/s, 2 objects/s recovering 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:52:27.217 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:52:27.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.15 deep-scrub starts 2024-03-26T21:52:27.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.15 deep-scrub ok 2024-03-26T21:52:27.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:52:27.218 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.e deep-scrub starts 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.e deep-scrub ok 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 56 B/s, 2 objects/s recovering 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:27.352 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.15 deep-scrub starts 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.15 deep-scrub ok 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.e deep-scrub starts 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.e deep-scrub ok 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 56 B/s, 2 objects/s recovering 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2239755953' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T21:52:27.353 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.15 deep-scrub starts 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.15 deep-scrub ok 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:52:27.354 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:27 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:52:27.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:28 smithi067 ceph-mon[33346]: 3.1f deep-scrub starts 2024-03-26T21:52:28.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:28 smithi067 ceph-mon[33346]: 3.1f deep-scrub ok 2024-03-26T21:52:28.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:28 smithi055 ceph-mon[40587]: 3.1f deep-scrub starts 2024-03-26T21:52:28.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:28 smithi055 ceph-mon[40587]: 3.1f deep-scrub ok 2024-03-26T21:52:28.525 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:28 smithi055 ceph-mon[29406]: 3.1f deep-scrub starts 2024-03-26T21:52:28.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:28 smithi055 ceph-mon[29406]: 3.1f deep-scrub ok 2024-03-26T21:52:28.544 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:28.544 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:29.111 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:29.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T21:52:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:29 smithi067 ceph-mon[33346]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 755 B/s rd, 0 op/s; 48 B/s, 2 objects/s recovering 2024-03-26T21:52:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:29 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:29 smithi067 ceph-mon[33346]: 2.0 scrub starts 2024-03-26T21:52:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:29 smithi067 ceph-mon[33346]: 4.4 deep-scrub starts 2024-03-26T21:52:29.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:29 smithi067 ceph-mon[33346]: 4.4 deep-scrub ok 2024-03-26T21:52:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[40587]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 755 B/s rd, 0 op/s; 48 B/s, 2 objects/s recovering 2024-03-26T21:52:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[40587]: 2.0 scrub starts 2024-03-26T21:52:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[40587]: 4.4 deep-scrub starts 2024-03-26T21:52:29.566 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[40587]: 4.4 deep-scrub ok 2024-03-26T21:52:29.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[29406]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 755 B/s rd, 0 op/s; 48 B/s, 2 objects/s recovering 2024-03-26T21:52:29.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:29.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[29406]: 2.0 scrub starts 2024-03-26T21:52:29.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[29406]: 4.4 deep-scrub starts 2024-03-26T21:52:29.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:29 smithi055 ceph-mon[29406]: 4.4 deep-scrub ok 2024-03-26T21:52:30.118 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T21:52:30.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[29406]: 2.0 scrub ok 2024-03-26T21:52:30.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:52:30.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:52:30.474 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:30.475 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[40587]: 2.0 scrub ok 2024-03-26T21:52:30.475 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:52:30.475 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:52:30.475 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:30 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:30 smithi067 ceph-mon[33346]: 2.0 scrub ok 2024-03-26T21:52:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:30 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:52:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:30 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:52:30.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:30 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:30.893 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-26T21:52:30.973 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:30.973 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:31.145 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:31.395 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[29406]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:52:31.395 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:31.395 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[29406]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:52:31.395 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[29406]: 3.5 scrub starts 2024-03-26T21:52:31.395 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[29406]: 3.5 scrub ok 2024-03-26T21:52:31.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[40587]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:52:31.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:31.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[40587]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:52:31.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[40587]: 3.5 scrub starts 2024-03-26T21:52:31.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:31 smithi055 ceph-mon[40587]: 3.5 scrub ok 2024-03-26T21:52:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:31 smithi067 ceph-mon[33346]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T21:52:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:31 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1626488715' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:31 smithi067 ceph-mon[33346]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T21:52:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:31 smithi067 ceph-mon[33346]: 3.5 scrub starts 2024-03-26T21:52:31.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:31 smithi067 ceph-mon[33346]: 3.5 scrub ok 2024-03-26T21:52:31.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T21:52:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:32 smithi067 ceph-mon[33346]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:33.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:32 smithi067 ceph-mon[33346]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:52:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:32 smithi055 ceph-mon[29406]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:32 smithi055 ceph-mon[29406]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:52:33.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:32 smithi055 ceph-mon[40587]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:52:33.319 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:32 smithi055 ceph-mon[40587]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T21:52:33.566 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:33.566 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:33.994 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-26T21:52:34.189 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: 2.2 scrub starts 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: 2.2 scrub ok 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1375384374' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: 6.6 deep-scrub starts 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: 6.6 deep-scrub ok 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:34.190 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:34 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:34.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: 2.2 scrub starts 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: 2.2 scrub ok 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1375384374' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: 6.6 deep-scrub starts 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: 6.6 deep-scrub ok 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:34.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: 2.2 scrub starts 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: 2.2 scrub ok 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1375384374' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: 6.6 deep-scrub starts 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: 6.6 deep-scrub ok 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:52:34.282 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:34 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:34.640 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T21:52:34.893 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:35 smithi067 ceph-mon[33346]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:35 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:35.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:35 smithi067 ceph-mon[33346]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:52:35.293 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[40587]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:35.293 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:35.293 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[40587]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:52:35.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[29406]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:35.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T21:52:35.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:35 smithi055 ceph-mon[29406]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T21:52:36.202 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:36.203 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:36.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T21:52:37.214 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:52:37.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-mon[33346]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:52:37.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[29406]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:52:37.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:37 smithi055 ceph-mon[40587]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T21:52:37.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:38.038 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: 6.1d deep-scrub starts 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: 6.1d deep-scrub ok 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2674639215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.314 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.315 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:52:38.315 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:38 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: 6.1d deep-scrub starts 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: 6.1d deep-scrub ok 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2674639215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: 6.1d deep-scrub starts 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: 6.1d deep-scrub ok 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2674639215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:52:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:52:38.734 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-26T21:52:38.883 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:38.883 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:38.982 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:39.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: 6.1 deep-scrub starts 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:52:39.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:39 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:52:39.246 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:39.246 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: 6.1 deep-scrub starts 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:39.247 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: 6.1 deep-scrub starts 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:52:39.248 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:39 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:52:39.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T21:52:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: 6.1 deep-scrub ok 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: 3.6 scrub starts 2024-03-26T21:52:40.176 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:40 smithi067 ceph-mon[33346]: 3.6 scrub ok 2024-03-26T21:52:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:52:40.316 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: 6.1 deep-scrub ok 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: 3.6 scrub starts 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[40587]: 3.6 scrub ok 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: 6.1 deep-scrub ok 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: 3.6 scrub starts 2024-03-26T21:52:40.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:40 smithi055 ceph-mon[29406]: 3.6 scrub ok 2024-03-26T21:52:41.067 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[40587]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2036594399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:52:41.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[29406]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:52:41.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2036594399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:41 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.333 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:41 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:52:41.333 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:41 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:52:41.333 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:41 smithi067 ceph-mon[33346]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T21:52:41.333 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:41 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2036594399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.333 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:41 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T21:52:41.375 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:41.375 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:41.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T21:52:42.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T21:52:42.141 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:42 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:52:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:42 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:52:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:42 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:42 smithi067 ceph-mon[33346]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:52:42.240 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:42 smithi067 ceph-mon[33346]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[29406]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[29406]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:52:42.263 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:52:42.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:52:42.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T21:52:42.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[40587]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T21:52:42.264 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:42 smithi055 ceph-mon[40587]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T21:52:42.268 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: 6.13 scrub starts 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: 6.13 scrub ok 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: 3.3 deep-scrub starts 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: 3.3 deep-scrub ok 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[29406]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: 6.13 scrub starts 2024-03-26T21:52:43.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: 6.13 scrub ok 2024-03-26T21:52:43.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: 3.3 deep-scrub starts 2024-03-26T21:52:43.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: 3.3 deep-scrub ok 2024-03-26T21:52:43.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:43.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:43.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:43 smithi055 ceph-mon[40587]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:52:43.370 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: 6.13 scrub starts 2024-03-26T21:52:43.370 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: 6.13 scrub ok 2024-03-26T21:52:43.370 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: 3.3 deep-scrub starts 2024-03-26T21:52:43.370 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: 3.3 deep-scrub ok 2024-03-26T21:52:43.371 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:43.371 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:43.371 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:43 smithi067 ceph-mon[33346]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:52:43.874 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:43.874 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:44.487 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:44 smithi067 ceph-mon[33346]: 4.7 scrub starts 2024-03-26T21:52:44.487 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:44 smithi067 ceph-mon[33346]: 4.7 scrub ok 2024-03-26T21:52:44.487 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:44 smithi067 ceph-mon[33346]: 3.d scrub starts 2024-03-26T21:52:44.487 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:44 smithi067 ceph-mon[33346]: 3.d scrub ok 2024-03-26T21:52:44.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T21:52:44.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[29406]: 4.7 scrub starts 2024-03-26T21:52:44.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[29406]: 4.7 scrub ok 2024-03-26T21:52:44.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[29406]: 3.d scrub starts 2024-03-26T21:52:44.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[29406]: 3.d scrub ok 2024-03-26T21:52:44.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[40587]: 4.7 scrub starts 2024-03-26T21:52:44.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[40587]: 4.7 scrub ok 2024-03-26T21:52:44.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[40587]: 3.d scrub starts 2024-03-26T21:52:44.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:44 smithi055 ceph-mon[40587]: 3.d scrub ok 2024-03-26T21:52:44.763 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:45.102 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T21:52:45.181 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:52:45.181 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:52:45.181 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:52:45.182 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: 5.14 scrub starts 2024-03-26T21:52:45.182 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: 5.14 scrub ok 2024-03-26T21:52:45.182 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3529347117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.182 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:45 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: 5.14 scrub starts 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: 5.14 scrub ok 2024-03-26T21:52:45.407 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3529347117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: pgmap v1146: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: 5.14 scrub starts 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: 5.14 scrub ok 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3529347117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.408 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:45 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:45.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T21:52:46.111 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:46.380 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:46.380 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:46.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:46 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:46.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:46 smithi067 ceph-mon[33346]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:52:46.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:46 smithi067 ceph-mon[33346]: 4.5 deep-scrub starts 2024-03-26T21:52:46.381 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:46 smithi067 ceph-mon[33346]: 4.5 deep-scrub ok 2024-03-26T21:52:46.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:46.443 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[40587]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[40587]: 4.5 deep-scrub starts 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[40587]: 4.5 deep-scrub ok 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[29406]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[29406]: 4.5 deep-scrub starts 2024-03-26T21:52:46.444 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:46 smithi055 ceph-mon[29406]: 4.5 deep-scrub ok 2024-03-26T21:52:47.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T21:52:47.214 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[40587]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:47.214 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:52:47.214 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:52:47.214 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[40587]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:52:47.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[29406]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:47.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:52:47.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:52:47.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:47 smithi055 ceph-mon[29406]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:52:47.310 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:47.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:47 smithi067 ceph-mon[33346]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:47.431 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:47 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:52:47.432 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:47 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:52:47.432 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:47 smithi067 ceph-mon[33346]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T21:52:47.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:48.139 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T21:52:48.467 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:48.467 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:48.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:48.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:48.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T21:52:48.491 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:52:48.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T21:52:49.034 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:49.034 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:49.102 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:49.145 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:49.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:49 smithi067 ceph-mon[33346]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:49.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:49.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:49 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:49.400 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:49 smithi067 ceph-mon[33346]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[29406]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[29406]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[40587]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/687537156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T21:52:49.505 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:49 smithi055 ceph-mon[40587]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T21:52:49.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T21:52:50.051 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[29406]: 3.e scrub starts 2024-03-26T21:52:50.316 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[29406]: 3.e scrub ok 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[29406]: 6.1e scrub starts 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[29406]: 6.1e scrub ok 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[40587]: 3.e scrub starts 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[40587]: 3.e scrub ok 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[40587]: 6.1e scrub starts 2024-03-26T21:52:50.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:50 smithi055 ceph-mon[40587]: 6.1e scrub ok 2024-03-26T21:52:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:50 smithi067 ceph-mon[33346]: 3.e scrub starts 2024-03-26T21:52:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:50 smithi067 ceph-mon[33346]: 3.e scrub ok 2024-03-26T21:52:50.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:50 smithi067 ceph-mon[33346]: 6.1e scrub starts 2024-03-26T21:52:50.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:50 smithi067 ceph-mon[33346]: 6.1e scrub ok 2024-03-26T21:52:51.175 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T21:52:51.409 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:51.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 4.11 scrub starts 2024-03-26T21:52:51.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 4.11 scrub ok 2024-03-26T21:52:51.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 5.16 scrub starts 2024-03-26T21:52:51.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 5.16 scrub ok 2024-03-26T21:52:51.410 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 5.1f scrub starts 2024-03-26T21:52:51.411 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 5.1f scrub ok 2024-03-26T21:52:51.411 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 4.1d deep-scrub starts 2024-03-26T21:52:51.411 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: 4.1d deep-scrub ok 2024-03-26T21:52:51.411 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:51.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 4.11 scrub starts 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 4.11 scrub ok 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 5.16 scrub starts 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 5.16 scrub ok 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 5.1f scrub starts 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 5.1f scrub ok 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 4.1d deep-scrub starts 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: 4.1d deep-scrub ok 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:51.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 4.11 scrub starts 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 4.11 scrub ok 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 5.16 scrub starts 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 5.16 scrub ok 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 5.1f scrub starts 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 5.1f scrub ok 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 4.1d deep-scrub starts 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: 4.1d deep-scrub ok 2024-03-26T21:52:51.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:51.691 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:51.691 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:51.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-26T21:52:52.191 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:52.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T21:52:52.460 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:52.460 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:52 smithi055 ceph-mon[29406]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:52:52.461 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:52.461 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:52 smithi055 ceph-mon[40587]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:52:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2907974663' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:52.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:52 smithi067 ceph-mon[33346]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T21:52:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T21:52:53.344 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:53 smithi067 ceph-mon[33346]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:53.497 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:53 smithi055 ceph-mon[40587]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:53.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:53 smithi055 ceph-mon[29406]: pgmap v1154: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:52:54.207 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-26T21:52:54.247 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-26T21:52:54.247 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T21:52:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:54 smithi067 ceph-mon[33346]: 1.0 scrub starts 2024-03-26T21:52:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:54 smithi067 ceph-mon[33346]: 1.0 scrub ok 2024-03-26T21:52:54.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:54 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:54.528 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[40587]: 1.0 scrub starts 2024-03-26T21:52:54.528 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[40587]: 1.0 scrub ok 2024-03-26T21:52:54.528 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:54.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[29406]: 1.0 scrub starts 2024-03-26T21:52:54.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[29406]: 1.0 scrub ok 2024-03-26T21:52:54.529 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:54 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T21:52:54.848 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T21:52:55.094 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:55.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: 3.1 scrub starts 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: 3.1 scrub ok 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: 3.14 scrub starts 2024-03-26T21:52:55.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:55 smithi067 ceph-mon[33346]: 3.14 scrub ok 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: 3.1 scrub starts 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: 3.1 scrub ok 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: 3.14 scrub starts 2024-03-26T21:52:55.525 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[40587]: 3.14 scrub ok 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3060301201' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: 3.1 scrub starts 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: 3.1 scrub ok 2024-03-26T21:52:55.526 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: 3.14 scrub starts 2024-03-26T21:52:55.527 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:55 smithi055 ceph-mon[29406]: 3.14 scrub ok 2024-03-26T21:52:56.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:56 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:52:56.489 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:56 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:52:56.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:56 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:52:56.490 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:56 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:52:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:52:56.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:52:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:57 smithi067 ceph-mon[33346]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:57 smithi067 ceph-mon[33346]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:52:57.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:57 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:52:57.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[29406]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:57.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[29406]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:52:57.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:52:57.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[40587]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:52:57.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[40587]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T21:52:57.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:57 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:52:57.979 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T21:52:58.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:52:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:52:58.566 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[29406]: 5.1e scrub starts 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[29406]: 5.1e scrub ok 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[29406]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[40587]: 5.1e scrub starts 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[40587]: 5.1e scrub ok 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:52:58.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-mon[40587]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:52:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:58 smithi067 ceph-mon[33346]: 5.1e scrub starts 2024-03-26T21:52:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:58 smithi067 ceph-mon[33346]: 5.1e scrub ok 2024-03-26T21:52:58.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:58 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/340569649' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:52:58.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:58 smithi067 ceph-mon[33346]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T21:52:58.887 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-26T21:52:59.138 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:52:59.245 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:52:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:52:59.501 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[40587]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-26T21:52:59.501 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:59.501 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[40587]: 6.0 scrub starts 2024-03-26T21:52:59.501 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[40587]: 6.0 scrub ok 2024-03-26T21:52:59.502 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[29406]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-26T21:52:59.502 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:59.502 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[29406]: 6.0 scrub starts 2024-03-26T21:52:59.502 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:52:59 smithi055 ceph-mon[29406]: 6.0 scrub ok 2024-03-26T21:52:59.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:59 smithi067 ceph-mon[33346]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-26T21:52:59.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:52:59.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:59 smithi067 ceph-mon[33346]: 6.0 scrub starts 2024-03-26T21:52:59.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:52:59 smithi067 ceph-mon[33346]: 6.0 scrub ok 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[40587]: 3.9 scrub starts 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[40587]: 3.9 scrub ok 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[29406]: 3.9 scrub starts 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[29406]: 3.9 scrub ok 2024-03-26T21:53:00.552 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:00 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:00.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:00 smithi067 ceph-mon[33346]: 3.9 scrub starts 2024-03-26T21:53:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:00 smithi067 ceph-mon[33346]: 3.9 scrub ok 2024-03-26T21:53:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:00 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: 4.1a scrub starts 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: 4.1a scrub ok 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1108560424' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:01.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: 4.1a scrub starts 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: 4.1a scrub ok 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1108560424' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:01 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:01.579 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-26T21:53:01.740 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: 4.1a scrub starts 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: 4.1a scrub ok 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1108560424' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:01.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:01 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T21:53:02.367 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T21:53:02.624 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:02.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 3.10 scrub starts 2024-03-26T21:53:02.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 3.10 scrub ok 2024-03-26T21:53:02.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 6.5 scrub starts 2024-03-26T21:53:02.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 6.5 scrub ok 2024-03-26T21:53:02.742 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 6.1a deep-scrub starts 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: 6.1a deep-scrub ok 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[40587]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 3.10 scrub starts 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 3.10 scrub ok 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 6.5 scrub starts 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 6.5 scrub ok 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:53:02.743 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 6.1a deep-scrub starts 2024-03-26T21:53:02.744 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: 6.1a deep-scrub ok 2024-03-26T21:53:02.744 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:02 smithi055 ceph-mon[29406]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:02.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 3.10 scrub starts 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 3.10 scrub ok 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 6.5 scrub starts 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 6.5 scrub ok 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 6.1a deep-scrub starts 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: 6.1a deep-scrub ok 2024-03-26T21:53:02.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:02 smithi067 ceph-mon[33346]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:03.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:03 smithi055 ceph-mon[40587]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:53:03.863 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:03.863 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:03 smithi055 ceph-mon[29406]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:53:03.863 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:03.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:03 smithi067 ceph-mon[33346]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T21:53:03.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:04.582 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.1b scrub starts 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.1b scrub ok 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 6.14 scrub starts 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 6.14 scrub ok 2024-03-26T21:53:04.890 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: pgmap v1165: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.a scrub starts 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: 3.a scrub ok 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.1b scrub starts 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.1b scrub ok 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 6.14 scrub starts 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 6.14 scrub ok 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: pgmap v1165: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.a scrub starts 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: 3.a scrub ok 2024-03-26T21:53:04.891 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:04 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.1b scrub starts 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.1b scrub ok 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 6.14 scrub starts 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 6.14 scrub ok 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: pgmap v1165: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.a scrub starts 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: 3.a scrub ok 2024-03-26T21:53:04.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:04 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:05.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-26T21:53:05.476 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[40587]: 4.16 scrub starts 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[40587]: 4.16 scrub ok 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[40587]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[40587]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[29406]: 4.16 scrub starts 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[29406]: 4.16 scrub ok 2024-03-26T21:53:05.870 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[29406]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:53:05.871 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:05.871 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:05 smithi055 ceph-mon[29406]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:53:05.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:05 smithi067 ceph-mon[33346]: 4.16 scrub starts 2024-03-26T21:53:05.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:05 smithi067 ceph-mon[33346]: 4.16 scrub ok 2024-03-26T21:53:05.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:05 smithi067 ceph-mon[33346]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T21:53:05.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:05 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3220061362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:05.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:05 smithi067 ceph-mon[33346]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T21:53:06.886 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[40587]: 3.12 scrub starts 2024-03-26T21:53:06.886 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[40587]: 3.12 scrub ok 2024-03-26T21:53:06.886 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[40587]: pgmap v1167: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:06.886 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[29406]: 3.12 scrub starts 2024-03-26T21:53:06.886 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[29406]: 3.12 scrub ok 2024-03-26T21:53:06.887 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:06 smithi055 ceph-mon[29406]: pgmap v1167: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:06 smithi067 ceph-mon[33346]: 3.12 scrub starts 2024-03-26T21:53:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:06 smithi067 ceph-mon[33346]: 3.12 scrub ok 2024-03-26T21:53:06.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:06 smithi067 ceph-mon[33346]: pgmap v1167: 132 pgs: 8 peering, 124 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:07.995 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-26T21:53:08.240 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: 4.1c deep-scrub starts 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: 4.1c deep-scrub ok 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: 4.8 scrub starts 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: 4.8 scrub ok 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2520199700' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:07 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: 4.1c deep-scrub starts 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: 4.1c deep-scrub ok 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: 4.8 scrub starts 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: 4.8 scrub ok 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2520199700' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: 4.1c deep-scrub starts 2024-03-26T21:53:08.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: 4.1c deep-scrub ok 2024-03-26T21:53:08.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: 4.8 scrub starts 2024-03-26T21:53:08.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: 4.8 scrub ok 2024-03-26T21:53:08.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2520199700' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:07 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T21:53:08.818 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 3.0 scrub starts 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 3.0 scrub ok 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 6.a scrub starts 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 6.a scrub ok 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:09.067 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: Cluster is now healthy 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 5.1a scrub starts 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[40587]: 5.1a scrub ok 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 3.0 scrub starts 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 3.0 scrub ok 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 6.a scrub starts 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 6.a scrub ok 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-26T21:53:09.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: Cluster is now healthy 2024-03-26T21:53:09.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:09.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:53:09.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 5.1a scrub starts 2024-03-26T21:53:09.069 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:08 smithi055 ceph-mon[29406]: 5.1a scrub ok 2024-03-26T21:53:09.069 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 3.0 scrub starts 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 3.0 scrub ok 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 6.a scrub starts 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 6.a scrub ok 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: Cluster is now healthy 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 5.1a scrub starts 2024-03-26T21:53:09.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:08 smithi067 ceph-mon[33346]: 5.1a scrub ok 2024-03-26T21:53:11.003 INFO:teuthology.orchestra.run.smithi055.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[40587]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/3934918370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[29406]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/3934918370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:11 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:11 smithi067 ceph-mon[33346]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T21:53:11.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:11 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/3934918370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:11 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:11.709 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-26T21:53:11.959 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: 4.9 scrub starts 2024-03-26T21:53:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: 4.9 scrub ok 2024-03-26T21:53:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:53:12.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: 3.c scrub starts 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[40587]: 3.c scrub ok 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: 4.9 scrub starts 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: 4.9 scrub ok 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: 3.c scrub starts 2024-03-26T21:53:12.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:12 smithi055 ceph-mon[29406]: 3.c scrub ok 2024-03-26T21:53:12.321 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: 4.9 scrub starts 2024-03-26T21:53:12.321 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: 4.9 scrub ok 2024-03-26T21:53:12.321 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:12.321 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T21:53:12.322 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: 3.c scrub starts 2024-03-26T21:53:12.322 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:12 smithi067 ceph-mon[33346]: 3.c scrub ok 2024-03-26T21:53:13.283 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:13 smithi067 ceph-mon[33346]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T21:53:13.288 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:13 smithi067 ceph-mon[33346]: 3.b scrub starts 2024-03-26T21:53:13.288 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:13 smithi067 ceph-mon[33346]: 3.b scrub ok 2024-03-26T21:53:13.288 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:13 smithi067 ceph-mon[33346]: 6.16 scrub starts 2024-03-26T21:53:13.288 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:13 smithi067 ceph-mon[33346]: 6.16 scrub ok 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[40587]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[40587]: 3.b scrub starts 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[40587]: 3.b scrub ok 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[40587]: 6.16 scrub starts 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[40587]: 6.16 scrub ok 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[29406]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[29406]: 3.b scrub starts 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[29406]: 3.b scrub ok 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[29406]: 6.16 scrub starts 2024-03-26T21:53:13.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:13 smithi055 ceph-mon[29406]: 6.16 scrub ok 2024-03-26T21:53:14.037 INFO:teuthology.orchestra.run.smithi055.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[29406]: 6.4 scrub starts 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[29406]: 6.4 scrub ok 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2135181694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[40587]: 6.4 scrub starts 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[40587]: 6.4 scrub ok 2024-03-26T21:53:14.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2135181694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:14 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:14 smithi067 ceph-mon[33346]: 6.4 scrub starts 2024-03-26T21:53:14.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:14 smithi067 ceph-mon[33346]: 6.4 scrub ok 2024-03-26T21:53:14.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:14 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2135181694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:14 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T21:53:14.757 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-26T21:53:14.757 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-26T21:53:14.757 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-26T21:53:14.757 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-26T21:53:14.757 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T21:53:14.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:53:15.004 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[40587]: 6.8 scrub starts 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[40587]: 6.8 scrub ok 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[40587]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[40587]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[40587]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[29406]: 6.8 scrub starts 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[29406]: 6.8 scrub ok 2024-03-26T21:53:15.294 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[29406]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:15.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[29406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:15.295 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:15 smithi055 ceph-mon[29406]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:53:15.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:15 smithi067 ceph-mon[33346]: 6.8 scrub starts 2024-03-26T21:53:15.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:15 smithi067 ceph-mon[33346]: 6.8 scrub ok 2024-03-26T21:53:15.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:15 smithi067 ceph-mon[33346]: pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:15.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:15 smithi067 ceph-mon[33346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T21:53:15.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:15 smithi067 ceph-mon[33346]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[40587]: 5.c scrub starts 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[40587]: 5.c scrub ok 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[40587]: 3.11 deep-scrub starts 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[40587]: 3.11 deep-scrub ok 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[29406]: 5.c scrub starts 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[29406]: 5.c scrub ok 2024-03-26T21:53:16.280 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[29406]: 3.11 deep-scrub starts 2024-03-26T21:53:16.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:16 smithi055 ceph-mon[29406]: 3.11 deep-scrub ok 2024-03-26T21:53:16.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:16 smithi067 ceph-mon[33346]: 5.c scrub starts 2024-03-26T21:53:16.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:16 smithi067 ceph-mon[33346]: 5.c scrub ok 2024-03-26T21:53:16.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:16 smithi067 ceph-mon[33346]: 3.11 deep-scrub starts 2024-03-26T21:53:16.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:16 smithi067 ceph-mon[33346]: 3.11 deep-scrub ok 2024-03-26T21:53:17.069 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:53:17.070 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":534,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","created":"2024-03-26T21:22:56.673257+0000","modified":"2024-03-26T21:53:14.030854+0000","last_up_change":"2024-03-26T21:41:24.572964+0000","last_in_change":"2024-03-26T21:52:07.278374+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:26:13.884991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"484","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T21:28:33.142617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T21:28:33.591599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:28:34.904126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:28:36.852373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:28:38.937273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f487071-c151-406b-b25b-3135cb3c63ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":317381212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":317381212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":317381212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":317381212}]},"public_addr":"172.21.15.55:6801/317381212","cluster_addr":"172.21.15.55:6802/317381212","heartbeat_back_addr":"172.21.15.55:6804/317381212","heartbeat_front_addr":"172.21.15.55:6803/317381212","state":["exists","up"]},{"osd":1,"uuid":"4fbc90fe-926a-4ac6-b5f0-d483ba4c09db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":3733554168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3733554168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3733554168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":3733554168}]},"public_addr":"172.21.15.55:6805/3733554168","cluster_addr":"172.21.15.55:6806/3733554168","heartbeat_back_addr":"172.21.15.55:6808/3733554168","heartbeat_front_addr":"172.21.15.55:6807/3733554168","state":["exists","up"]},{"osd":2,"uuid":"e2fe8018-1060-499c-a7e6-56d4da0eba7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6809","nonce":3059242923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3059242923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3059242923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6811","nonce":3059242923}]},"public_addr":"172.21.15.55:6809/3059242923","cluster_addr":"172.21.15.55:6810/3059242923","heartbeat_back_addr":"172.21.15.55:6812/3059242923","heartbeat_front_addr":"172.21.15.55:6811/3059242923","state":["exists","up"]},{"osd":3,"uuid":"30637a8d-5c1b-4ede-8214-a4595c875d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6813","nonce":3851321468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3851321468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3851321468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6815","nonce":3851321468}]},"public_addr":"172.21.15.55:6813/3851321468","cluster_addr":"172.21.15.55:6814/3851321468","heartbeat_back_addr":"172.21.15.55:6816/3851321468","heartbeat_front_addr":"172.21.15.55:6815/3851321468","state":["exists","up"]},{"osd":4,"uuid":"cd2d68be-4b10-4242-826b-e085c2ce5598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":92,"up_from":133,"up_thru":509,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":254752278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6801","nonce":254752278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":254752278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":254752278}]},"public_addr":"172.21.15.67:6800/254752278","cluster_addr":"172.21.15.67:6801/254752278","heartbeat_back_addr":"172.21.15.67:6803/254752278","heartbeat_front_addr":"172.21.15.67:6802/254752278","state":["exists","up"]},{"osd":5,"uuid":"7468b94a-3216-4bc2-9b57-1d93e7c0b213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":278711358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":278711358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":278711358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":278711358}]},"public_addr":"172.21.15.67:6804/278711358","cluster_addr":"172.21.15.67:6805/278711358","heartbeat_back_addr":"172.21.15.67:6807/278711358","heartbeat_front_addr":"172.21.15.67:6806/278711358","state":["exists","up"]},{"osd":6,"uuid":"d47c79f3-96bf-46b6-a7f3-a315446d04b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":529,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3926251639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":3926251639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3926251639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3926251639}]},"public_addr":"172.21.15.67:6808/3926251639","cluster_addr":"172.21.15.67:6809/3926251639","heartbeat_back_addr":"172.21.15.67:6811/3926251639","heartbeat_front_addr":"172.21.15.67:6810/3926251639","state":["exists","up"]},{"osd":7,"uuid":"f10b98b8-887d-4f83-8bc9-99f5c302dc55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":248,"up_from":276,"up_thru":504,"down_at":249,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":650373445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":650373445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":650373445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":650373445}]},"public_addr":"172.21.15.67:6812/650373445","cluster_addr":"172.21.15.67:6813/650373445","heartbeat_back_addr":"172.21.15.67:6815/650373445","heartbeat_front_addr":"172.21.15.67:6814/650373445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:41.785012+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:15.101533+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:38.085232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:36.902939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:37.497921+0000","dead_epoch":92},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:55.471476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:09.564754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:21.143768+0000","dead_epoch":248}],"pg_upmap":[{"pgid":"5.17","osds":[1,2,3]},{"pgid":"5.1a","osds":[0,1,3]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":0}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":3,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.19","mappings":[{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2649740776":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/4141219490":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/147833426":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2854772543":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/2804789175":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/2199694469":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/1614234257":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/2834138692":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/1740219825":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/268664696":"2024-03-27T21:23:57.787077+0000","172.21.15.55:6800/536717527":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/966571165":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2757111431":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/559658018":"2024-03-27T21:23:57.787077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.0 scrub starts 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.0 scrub ok 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.17 scrub starts 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.17 scrub ok 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.9 deep-scrub starts 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: 5.9 deep-scrub ok 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[29406]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.0 scrub starts 2024-03-26T21:53:17.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.0 scrub ok 2024-03-26T21:53:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.17 scrub starts 2024-03-26T21:53:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.17 scrub ok 2024-03-26T21:53:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.9 deep-scrub starts 2024-03-26T21:53:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: 5.9 deep-scrub ok 2024-03-26T21:53:17.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:17 smithi055 ceph-mon[40587]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.0 scrub starts 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.0 scrub ok 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.17 scrub starts 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.17 scrub ok 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.9 deep-scrub starts 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: 5.9 deep-scrub ok 2024-03-26T21:53:17.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-mon[33346]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:17.769 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T21:53:17.769 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T21:53:18.015 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:18.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:18 smithi067 ceph-mon[33346]: 3.7 scrub starts 2024-03-26T21:53:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:18 smithi067 ceph-mon[33346]: 3.7 scrub ok 2024-03-26T21:53:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:18 smithi067 ceph-mon[33346]: 5.b scrub starts 2024-03-26T21:53:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:18 smithi067 ceph-mon[33346]: 5.b scrub ok 2024-03-26T21:53:18.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:18 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2338839880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:18.324 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[40587]: 3.7 scrub starts 2024-03-26T21:53:18.324 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[40587]: 3.7 scrub ok 2024-03-26T21:53:18.324 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[40587]: 5.b scrub starts 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[40587]: 5.b scrub ok 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2338839880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[29406]: 3.7 scrub starts 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[29406]: 3.7 scrub ok 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[29406]: 5.b scrub starts 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[29406]: 5.b scrub ok 2024-03-26T21:53:18.325 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2338839880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:19.214 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:18 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:19.214 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: 4.d scrub starts 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: 4.d scrub ok 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: 4.1f scrub starts 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[40587]: 4.1f scrub ok 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: 4.d scrub starts 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: 4.d scrub ok 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: 4.1f scrub starts 2024-03-26T21:53:19.215 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:19 smithi055 ceph-mon[29406]: 4.1f scrub ok 2024-03-26T21:53:19.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: 4.d scrub starts 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: 4.d scrub ok 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: pgmap v1176: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: 4.1f scrub starts 2024-03-26T21:53:19.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:19 smithi067 ceph-mon[33346]: 4.1f scrub ok 2024-03-26T21:53:19.993 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:53:19.993 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":534,"fsid":"e5f52b02-ebb6-11ee-95d0-87774f69a715","created":"2024-03-26T21:22:56.673257+0000","modified":"2024-03-26T21:53:14.030854+0000","last_up_change":"2024-03-26T21:41:24.572964+0000","last_in_change":"2024-03-26T21:52:07.278374+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T21:26:13.884991+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"484","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T21:28:33.142617+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T21:28:33.591599+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T21:28:34.904126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T21:28:36.852373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T21:28:38.937273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f487071-c151-406b-b25b-3135cb3c63ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6801","nonce":317381212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":317381212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":317381212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6803","nonce":317381212}]},"public_addr":"172.21.15.55:6801/317381212","cluster_addr":"172.21.15.55:6802/317381212","heartbeat_back_addr":"172.21.15.55:6804/317381212","heartbeat_front_addr":"172.21.15.55:6803/317381212","state":["exists","up"]},{"osd":1,"uuid":"4fbc90fe-926a-4ac6-b5f0-d483ba4c09db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6805","nonce":3733554168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3733554168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3733554168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6807","nonce":3733554168}]},"public_addr":"172.21.15.55:6805/3733554168","cluster_addr":"172.21.15.55:6806/3733554168","heartbeat_back_addr":"172.21.15.55:6808/3733554168","heartbeat_front_addr":"172.21.15.55:6807/3733554168","state":["exists","up"]},{"osd":2,"uuid":"e2fe8018-1060-499c-a7e6-56d4da0eba7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6809","nonce":3059242923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":3059242923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":3059242923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6811","nonce":3059242923}]},"public_addr":"172.21.15.55:6809/3059242923","cluster_addr":"172.21.15.55:6810/3059242923","heartbeat_back_addr":"172.21.15.55:6812/3059242923","heartbeat_front_addr":"172.21.15.55:6811/3059242923","state":["exists","up"]},{"osd":3,"uuid":"30637a8d-5c1b-4ede-8214-a4595c875d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6813","nonce":3851321468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":3851321468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3851321468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6815","nonce":3851321468}]},"public_addr":"172.21.15.55:6813/3851321468","cluster_addr":"172.21.15.55:6814/3851321468","heartbeat_back_addr":"172.21.15.55:6816/3851321468","heartbeat_front_addr":"172.21.15.55:6815/3851321468","state":["exists","up"]},{"osd":4,"uuid":"cd2d68be-4b10-4242-826b-e085c2ce5598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":92,"up_from":133,"up_thru":509,"down_at":93,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":254752278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6801","nonce":254752278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":254752278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":254752278}]},"public_addr":"172.21.15.67:6800/254752278","cluster_addr":"172.21.15.67:6801/254752278","heartbeat_back_addr":"172.21.15.67:6803/254752278","heartbeat_front_addr":"172.21.15.67:6802/254752278","state":["exists","up"]},{"osd":5,"uuid":"7468b94a-3216-4bc2-9b57-1d93e7c0b213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":278711358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":278711358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6807","nonce":278711358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":278711358}]},"public_addr":"172.21.15.67:6804/278711358","cluster_addr":"172.21.15.67:6805/278711358","heartbeat_back_addr":"172.21.15.67:6807/278711358","heartbeat_front_addr":"172.21.15.67:6806/278711358","state":["exists","up"]},{"osd":6,"uuid":"d47c79f3-96bf-46b6-a7f3-a315446d04b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":529,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3926251639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":3926251639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3926251639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3926251639}]},"public_addr":"172.21.15.67:6808/3926251639","cluster_addr":"172.21.15.67:6809/3926251639","heartbeat_back_addr":"172.21.15.67:6811/3926251639","heartbeat_front_addr":"172.21.15.67:6810/3926251639","state":["exists","up"]},{"osd":7,"uuid":"f10b98b8-887d-4f83-8bc9-99f5c302dc55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":248,"up_from":276,"up_thru":504,"down_at":249,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":650373445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":650373445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":650373445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":650373445}]},"public_addr":"172.21.15.67:6812/650373445","cluster_addr":"172.21.15.67:6813/650373445","heartbeat_back_addr":"172.21.15.67:6815/650373445","heartbeat_front_addr":"172.21.15.67:6814/650373445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:41.785012+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:15.101533+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:38.085232+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:36.902939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:37.497921+0000","dead_epoch":92},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:55.471476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:09.564754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T21:52:21.143768+0000","dead_epoch":248}],"pg_upmap":[{"pgid":"5.17","osds":[1,2,3]},{"pgid":"5.1a","osds":[0,1,3]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":0}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":3,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.19","mappings":[{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/2649740776":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/4141219490":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/147833426":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2854772543":"2024-03-27T21:29:15.251747+0000","172.21.15.55:6800/2804789175":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/2199694469":"2024-03-27T21:23:57.787077+0000","172.21.15.55:0/1614234257":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/2834138692":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/1740219825":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/268664696":"2024-03-27T21:23:57.787077+0000","172.21.15.55:6800/536717527":"2024-03-27T21:23:30.740110+0000","172.21.15.55:0/966571165":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/2757111431":"2024-03-27T21:29:15.251747+0000","172.21.15.55:0/559658018":"2024-03-27T21:23:57.787077+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 5.d scrub starts 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 5.d scrub ok 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 5.2 scrub starts 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 5.2 scrub ok 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 6.19 scrub starts 2024-03-26T21:53:20.280 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: 6.19 scrub ok 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1491887550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 5.d scrub starts 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 5.d scrub ok 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 5.2 scrub starts 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 5.2 scrub ok 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 6.19 scrub starts 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: 6.19 scrub ok 2024-03-26T21:53:20.281 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:20 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1491887550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 5.d scrub starts 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 5.d scrub ok 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 5.2 scrub starts 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 5.2 scrub ok 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 6.19 scrub starts 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: 6.19 scrub ok 2024-03-26T21:53:20.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:20 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1491887550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T21:53:20.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T21:53:20.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T21:53:20.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T21:53:20.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T21:53:20.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T21:53:20.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T21:53:20.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T21:53:20.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T21:53:21.243 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.244 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.244 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.244 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.249 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.252 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.261 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.289 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[40587]: 4.1 deep-scrub starts 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[40587]: 4.1 deep-scrub ok 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[40587]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[29406]: 4.1 deep-scrub starts 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[29406]: 4.1 deep-scrub ok 2024-03-26T21:53:21.378 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:21 smithi055 ceph-mon[29406]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:53:21.490 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:21 smithi067 ceph-mon[33346]: 4.1 deep-scrub starts 2024-03-26T21:53:21.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:21 smithi067 ceph-mon[33346]: 4.1 deep-scrub ok 2024-03-26T21:53:21.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:21 smithi067 ceph-mon[33346]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 6.f scrub starts 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 6.12 scrub starts 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 6.12 scrub ok 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 6.f scrub ok 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 4.b scrub starts 2024-03-26T21:53:22.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[40587]: 4.b scrub ok 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 6.f scrub starts 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 6.12 scrub starts 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 6.12 scrub ok 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 6.f scrub ok 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 4.b scrub starts 2024-03-26T21:53:22.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:22 smithi055 ceph-mon[29406]: 4.b scrub ok 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 6.f scrub starts 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 6.12 scrub starts 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 6.12 scrub ok 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 6.f scrub ok 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 4.b scrub starts 2024-03-26T21:53:22.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:22 smithi067 ceph-mon[33346]: 4.b scrub ok 2024-03-26T21:53:23.361 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:23 smithi067 ceph-mon[33346]: 6.1c scrub starts 2024-03-26T21:53:23.361 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:23 smithi067 ceph-mon[33346]: 6.1c scrub ok 2024-03-26T21:53:23.361 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:23 smithi067 ceph-mon[33346]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:23.460 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[29406]: 6.1c scrub starts 2024-03-26T21:53:23.460 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[29406]: 6.1c scrub ok 2024-03-26T21:53:23.461 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[29406]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:23.461 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[40587]: 6.1c scrub starts 2024-03-26T21:53:23.461 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[40587]: 6.1c scrub ok 2024-03-26T21:53:23.461 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:23 smithi055 ceph-mon[40587]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 4.14 scrub starts 2024-03-26T21:53:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 4.14 scrub ok 2024-03-26T21:53:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 5.18 deep-scrub starts 2024-03-26T21:53:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 5.18 deep-scrub ok 2024-03-26T21:53:24.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 5.19 scrub starts 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[29406]: 5.19 scrub ok 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 4.14 scrub starts 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 4.14 scrub ok 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 5.18 deep-scrub starts 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 5.18 deep-scrub ok 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 5.19 scrub starts 2024-03-26T21:53:24.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:24 smithi055 ceph-mon[40587]: 5.19 scrub ok 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 4.14 scrub starts 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 4.14 scrub ok 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 5.18 deep-scrub starts 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 5.18 deep-scrub ok 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 5.19 scrub starts 2024-03-26T21:53:24.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:24 smithi067 ceph-mon[33346]: 5.19 scrub ok 2024-03-26T21:53:25.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 3.15 scrub starts 2024-03-26T21:53:25.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 3.15 scrub ok 2024-03-26T21:53:25.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:25.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 6.18 scrub starts 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 6.18 scrub ok 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 3.16 deep-scrub starts 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[40587]: 3.16 deep-scrub ok 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 3.15 scrub starts 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 3.15 scrub ok 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 6.18 scrub starts 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 6.18 scrub ok 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 3.16 deep-scrub starts 2024-03-26T21:53:25.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:25 smithi055 ceph-mon[29406]: 3.16 deep-scrub ok 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 3.15 scrub starts 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 3.15 scrub ok 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 6.18 scrub starts 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 6.18 scrub ok 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 3.16 deep-scrub starts 2024-03-26T21:53:25.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:25 smithi067 ceph-mon[33346]: 3.16 deep-scrub ok 2024-03-26T21:53:26.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:26 smithi055 ceph-mon[40587]: 5.8 scrub starts 2024-03-26T21:53:26.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:26 smithi055 ceph-mon[40587]: 5.8 scrub ok 2024-03-26T21:53:26.421 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:26 smithi055 ceph-mon[29406]: 5.8 scrub starts 2024-03-26T21:53:26.422 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:26 smithi055 ceph-mon[29406]: 5.8 scrub ok 2024-03-26T21:53:26.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:26 smithi067 ceph-mon[33346]: 5.8 scrub starts 2024-03-26T21:53:26.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:26 smithi067 ceph-mon[33346]: 5.8 scrub ok 2024-03-26T21:53:27.403 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[29406]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T21:53:27.404 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[29406]: 4.a scrub starts 2024-03-26T21:53:27.404 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[29406]: 4.a scrub ok 2024-03-26T21:53:27.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[40587]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T21:53:27.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[40587]: 4.a scrub starts 2024-03-26T21:53:27.404 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:27 smithi055 ceph-mon[40587]: 4.a scrub ok 2024-03-26T21:53:27.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:27 smithi067 ceph-mon[33346]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T21:53:27.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:27 smithi067 ceph-mon[33346]: 4.a scrub starts 2024-03-26T21:53:27.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:27 smithi067 ceph-mon[33346]: 4.a scrub ok 2024-03-26T21:53:28.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:27 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:28.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:28 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:28.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:28 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:28.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:28 smithi067 ceph-mon[33346]: 4.13 scrub starts 2024-03-26T21:53:28.241 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:28 smithi067 ceph-mon[33346]: 4.13 scrub ok 2024-03-26T21:53:28.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:28.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:28.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[29406]: 4.13 scrub starts 2024-03-26T21:53:28.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[29406]: 4.13 scrub ok 2024-03-26T21:53:28.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:28.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:28.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[40587]: 4.13 scrub starts 2024-03-26T21:53:28.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-mon[40587]: 4.13 scrub ok 2024-03-26T21:53:29.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[29406]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:29.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:29.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[29406]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:53:29.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[29406]: 3.1f scrub starts 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[29406]: 3.1f scrub ok 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:28 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[40587]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[40587]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[40587]: 3.1f scrub starts 2024-03-26T21:53:29.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:29 smithi055 ceph-mon[40587]: 3.1f scrub ok 2024-03-26T21:53:29.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:29 smithi067 ceph-mon[33346]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:29.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:29 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:29.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:29 smithi067 ceph-mon[33346]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T21:53:29.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:29 smithi067 ceph-mon[33346]: 3.1f scrub starts 2024-03-26T21:53:29.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:29 smithi067 ceph-mon[33346]: 3.1f scrub ok 2024-03-26T21:53:30.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:30 smithi067 ceph-mon[33346]: 4.f scrub starts 2024-03-26T21:53:30.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:30 smithi067 ceph-mon[33346]: 4.f scrub ok 2024-03-26T21:53:30.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:30 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:30.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:30 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[29406]: 4.f scrub starts 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[29406]: 4.f scrub ok 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[40587]: 4.f scrub starts 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[40587]: 4.f scrub ok 2024-03-26T21:53:30.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:30.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:30 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 5.6 scrub starts 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 5.6 scrub ok 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 6.1b scrub starts 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 6.9 scrub starts 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 6.1b scrub ok 2024-03-26T21:53:31.420 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[29406]: 6.9 scrub ok 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 5.6 scrub starts 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 5.6 scrub ok 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 6.1b scrub starts 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 6.9 scrub starts 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 6.1b scrub ok 2024-03-26T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:31 smithi055 ceph-mon[40587]: 6.9 scrub ok 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 5.6 scrub starts 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 5.6 scrub ok 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 6.1b scrub starts 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 6.9 scrub starts 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 6.1b scrub ok 2024-03-26T21:53:31.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:31 smithi067 ceph-mon[33346]: 6.9 scrub ok 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 3.19 scrub starts 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 3.19 scrub ok 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 6.11 scrub starts 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 6.11 scrub ok 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 6.e scrub starts 2024-03-26T21:53:32.449 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[40587]: 6.e scrub ok 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 3.19 scrub starts 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 3.19 scrub ok 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 6.11 scrub starts 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 6.11 scrub ok 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 6.e scrub starts 2024-03-26T21:53:32.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:32 smithi055 ceph-mon[29406]: 6.e scrub ok 2024-03-26T21:53:32.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 3.19 scrub starts 2024-03-26T21:53:32.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 3.19 scrub ok 2024-03-26T21:53:32.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 6.11 scrub starts 2024-03-26T21:53:32.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 6.11 scrub ok 2024-03-26T21:53:32.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 6.e scrub starts 2024-03-26T21:53:32.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:32 smithi067 ceph-mon[33346]: 6.e scrub ok 2024-03-26T21:53:33.083 INFO:teuthology.orchestra.run.smithi055.stdout:55834575184 2024-03-26T21:53:33.083 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T21:53:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: pgmap v1185: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: 5.3 scrub starts 2024-03-26T21:53:33.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: 5.3 scrub ok 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: 6.17 scrub starts 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: 6.17 scrub ok 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: pgmap v1185: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: 5.3 scrub starts 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: 5.3 scrub ok 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: 6.17 scrub starts 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: 6.17 scrub ok 2024-03-26T21:53:33.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:33 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: pgmap v1185: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: 5.3 scrub starts 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: 5.3 scrub ok 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: 6.17 scrub starts 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: 6.17 scrub ok 2024-03-26T21:53:33.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:33 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 2.0 deep-scrub starts 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 2.0 deep-scrub ok 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 3.2 scrub starts 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 3.2 scrub ok 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 4.18 scrub starts 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[29406]: 4.18 scrub ok 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 2.0 deep-scrub starts 2024-03-26T21:53:34.450 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 2.0 deep-scrub ok 2024-03-26T21:53:34.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 3.2 scrub starts 2024-03-26T21:53:34.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 3.2 scrub ok 2024-03-26T21:53:34.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 4.18 scrub starts 2024-03-26T21:53:34.451 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:34 smithi055 ceph-mon[40587]: 4.18 scrub ok 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 2.0 deep-scrub starts 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 2.0 deep-scrub ok 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 3.2 scrub starts 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 3.2 scrub ok 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 4.18 scrub starts 2024-03-26T21:53:34.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:34 smithi067 ceph-mon[33346]: 4.18 scrub ok 2024-03-26T21:53:35.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[40587]: pgmap v1186: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:35.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[40587]: 6.b deep-scrub starts 2024-03-26T21:53:35.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[40587]: 6.b deep-scrub ok 2024-03-26T21:53:35.313 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[40587]: 5.1 scrub starts 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[40587]: 5.1 scrub ok 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[29406]: pgmap v1186: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[29406]: 6.b deep-scrub starts 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[29406]: 6.b deep-scrub ok 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[29406]: 5.1 scrub starts 2024-03-26T21:53:35.314 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:35 smithi055 ceph-mon[29406]: 5.1 scrub ok 2024-03-26T21:53:35.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:35 smithi067 ceph-mon[33346]: pgmap v1186: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:35.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:35 smithi067 ceph-mon[33346]: 6.b deep-scrub starts 2024-03-26T21:53:35.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:35 smithi067 ceph-mon[33346]: 6.b deep-scrub ok 2024-03-26T21:53:35.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:35 smithi067 ceph-mon[33346]: 5.1 scrub starts 2024-03-26T21:53:35.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:35 smithi067 ceph-mon[33346]: 5.1 scrub ok 2024-03-26T21:53:36.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 4.12 scrub starts 2024-03-26T21:53:36.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 4.12 scrub ok 2024-03-26T21:53:36.396 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 2.1 deep-scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 2.1 deep-scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 4.2 scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 4.2 scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 6.10 scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 6.10 scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 3.1d scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[40587]: 3.1d scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 4.12 scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 4.12 scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 2.1 deep-scrub starts 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 2.1 deep-scrub ok 2024-03-26T21:53:36.397 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 4.2 scrub starts 2024-03-26T21:53:36.398 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 4.2 scrub ok 2024-03-26T21:53:36.398 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 6.10 scrub starts 2024-03-26T21:53:36.398 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 6.10 scrub ok 2024-03-26T21:53:36.398 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 3.1d scrub starts 2024-03-26T21:53:36.398 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:36 smithi055 ceph-mon[29406]: 3.1d scrub ok 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 4.12 scrub starts 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 4.12 scrub ok 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 2.1 deep-scrub starts 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 2.1 deep-scrub ok 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 4.2 scrub starts 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 4.2 scrub ok 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 6.10 scrub starts 2024-03-26T21:53:36.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 6.10 scrub ok 2024-03-26T21:53:36.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 3.1d scrub starts 2024-03-26T21:53:36.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:36 smithi067 ceph-mon[33346]: 3.1d scrub ok 2024-03-26T21:53:37.171 INFO:teuthology.orchestra.run.smithi055.stdout:34359738710 2024-03-26T21:53:37.172 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T21:53:37.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-mon[33346]: 6.15 scrub starts 2024-03-26T21:53:37.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-mon[33346]: 6.15 scrub ok 2024-03-26T21:53:37.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-mon[33346]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:37.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-mon[33346]: 4.e scrub starts 2024-03-26T21:53:37.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-mon[33346]: 4.e scrub ok 2024-03-26T21:53:37.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[29406]: 6.15 scrub starts 2024-03-26T21:53:37.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[29406]: 6.15 scrub ok 2024-03-26T21:53:37.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[29406]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:37.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[29406]: 4.e scrub starts 2024-03-26T21:53:37.530 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[29406]: 4.e scrub ok 2024-03-26T21:53:37.531 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[40587]: 6.15 scrub starts 2024-03-26T21:53:37.531 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[40587]: 6.15 scrub ok 2024-03-26T21:53:37.531 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[40587]: pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:37.531 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[40587]: 4.e scrub starts 2024-03-26T21:53:37.531 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:37 smithi055 ceph-mon[40587]: 4.e scrub ok 2024-03-26T21:53:37.561 INFO:teuthology.orchestra.run.smithi055.stdout:571230650617 2024-03-26T21:53:37.561 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T21:53:38.068 INFO:teuthology.orchestra.run.smithi055.stdout:77309411658 2024-03-26T21:53:38.068 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T21:53:38.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:37 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[29406]: 4.c scrub starts 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[29406]: 4.c scrub ok 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[29406]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:38.317 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[40587]: 4.c scrub starts 2024-03-26T21:53:38.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[40587]: 4.c scrub ok 2024-03-26T21:53:38.318 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-mon[40587]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:53:38.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:38 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:38.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:38 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:38.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:38 smithi067 ceph-mon[33346]: 4.c scrub starts 2024-03-26T21:53:38.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:38 smithi067 ceph-mon[33346]: 4.c scrub ok 2024-03-26T21:53:38.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:38 smithi067 ceph-mon[33346]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T21:53:38.723 INFO:teuthology.orchestra.run.smithi055.stdout:214748365111 2024-03-26T21:53:38.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T21:53:39.233 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: 5.e scrub starts 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: 5.e scrub ok 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: 5.7 deep-scrub starts 2024-03-26T21:53:39.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:39 smithi067 ceph-mon[33346]: 5.7 deep-scrub ok 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: 5.e scrub starts 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: 5.e scrub ok 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: 5.7 deep-scrub starts 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[40587]: 5.7 deep-scrub ok 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:39.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:39.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: 5.e scrub starts 2024-03-26T21:53:39.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: 5.e scrub ok 2024-03-26T21:53:39.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: 5.7 deep-scrub starts 2024-03-26T21:53:39.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:39 smithi055 ceph-mon[29406]: 5.7 deep-scrub ok 2024-03-26T21:53:39.712 INFO:teuthology.orchestra.run.smithi055.stdout:111669150022 2024-03-26T21:53:39.713 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[29406]: 4.0 scrub starts 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[29406]: 4.0 scrub ok 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[29406]: 5.1c scrub starts 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[29406]: 5.1c scrub ok 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[40587]: 4.0 scrub starts 2024-03-26T21:53:40.543 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[40587]: 4.0 scrub ok 2024-03-26T21:53:40.544 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[40587]: 5.1c scrub starts 2024-03-26T21:53:40.544 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:40 smithi055 ceph-mon[40587]: 5.1c scrub ok 2024-03-26T21:53:40.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:40 smithi067 ceph-mon[33346]: 4.0 scrub starts 2024-03-26T21:53:40.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:40 smithi067 ceph-mon[33346]: 4.0 scrub ok 2024-03-26T21:53:40.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:40 smithi067 ceph-mon[33346]: 5.1c scrub starts 2024-03-26T21:53:40.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:40 smithi067 ceph-mon[33346]: 5.1c scrub ok 2024-03-26T21:53:40.870 INFO:teuthology.orchestra.run.smithi055.stdout:176093659451 2024-03-26T21:53:40.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T21:53:41.049 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.085 INFO:teuthology.orchestra.run.smithi055.stdout:1185410973843 2024-03-26T21:53:41.086 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T21:53:41.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[29406]: 5.4 scrub starts 2024-03-26T21:53:41.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[29406]: 5.4 scrub ok 2024-03-26T21:53:41.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[29406]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:41.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[29406]: 3.17 scrub starts 2024-03-26T21:53:41.386 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[29406]: 3.17 scrub ok 2024-03-26T21:53:41.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[40587]: 5.4 scrub starts 2024-03-26T21:53:41.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[40587]: 5.4 scrub ok 2024-03-26T21:53:41.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[40587]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:41.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[40587]: 3.17 scrub starts 2024-03-26T21:53:41.387 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:41 smithi055 ceph-mon[40587]: 3.17 scrub ok 2024-03-26T21:53:41.395 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.395 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.396 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.397 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.399 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:41.642 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:41 smithi067 ceph-mon[33346]: 5.4 scrub starts 2024-03-26T21:53:41.642 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:41 smithi067 ceph-mon[33346]: 5.4 scrub ok 2024-03-26T21:53:41.642 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:41 smithi067 ceph-mon[33346]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T21:53:41.642 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:41 smithi067 ceph-mon[33346]: 3.17 scrub starts 2024-03-26T21:53:41.642 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:41 smithi067 ceph-mon[33346]: 3.17 scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 5.10 deep-scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 5.10 deep-scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 5.1d scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 5.1d scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 4.6 scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 4.6 scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 4.15 scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 4.15 scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 3.1e scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[40587]: 3.1e scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 5.10 deep-scrub starts 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 5.10 deep-scrub ok 2024-03-26T21:53:42.548 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 5.1d scrub starts 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 5.1d scrub ok 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 4.6 scrub starts 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 4.6 scrub ok 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 4.15 scrub starts 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 4.15 scrub ok 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 3.1e scrub starts 2024-03-26T21:53:42.549 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:42 smithi055 ceph-mon[29406]: 3.1e scrub ok 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 5.10 deep-scrub starts 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 5.10 deep-scrub ok 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 5.1d scrub starts 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 5.1d scrub ok 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 4.6 scrub starts 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 4.6 scrub ok 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 4.15 scrub starts 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 4.15 scrub ok 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 3.1e scrub starts 2024-03-26T21:53:42.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:42 smithi067 ceph-mon[33346]: 3.1e scrub ok 2024-03-26T21:53:43.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[40587]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:43.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[40587]: 4.19 scrub starts 2024-03-26T21:53:43.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[40587]: 4.19 scrub ok 2024-03-26T21:53:43.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[29406]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:43.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[29406]: 4.19 scrub starts 2024-03-26T21:53:43.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:43 smithi055 ceph-mon[29406]: 4.19 scrub ok 2024-03-26T21:53:43.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:43 smithi067 ceph-mon[33346]: pgmap v1191: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:43.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:43 smithi067 ceph-mon[33346]: 4.19 scrub starts 2024-03-26T21:53:43.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:43 smithi067 ceph-mon[33346]: 4.19 scrub ok 2024-03-26T21:53:44.542 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:44 smithi055 ceph-mon[40587]: 4.1b scrub starts 2024-03-26T21:53:44.542 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:44 smithi055 ceph-mon[40587]: 4.1b scrub ok 2024-03-26T21:53:44.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:44 smithi055 ceph-mon[29406]: 4.1b scrub starts 2024-03-26T21:53:44.542 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:44 smithi055 ceph-mon[29406]: 4.1b scrub ok 2024-03-26T21:53:44.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:44 smithi067 ceph-mon[33346]: 4.1b scrub starts 2024-03-26T21:53:44.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:44 smithi067 ceph-mon[33346]: 4.1b scrub ok 2024-03-26T21:53:45.265 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:45.265 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[29406]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[29406]: 4.1e scrub starts 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[29406]: 4.1e scrub ok 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[40587]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[40587]: 4.1e scrub starts 2024-03-26T21:53:45.527 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:45 smithi055 ceph-mon[40587]: 4.1e scrub ok 2024-03-26T21:53:45.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:45 smithi067 ceph-mon[33346]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:45.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:45 smithi067 ceph-mon[33346]: 4.1e scrub starts 2024-03-26T21:53:45.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:45 smithi067 ceph-mon[33346]: 4.1e scrub ok 2024-03-26T21:53:46.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:46 smithi055 ceph-mon[29406]: 3.6 deep-scrub starts 2024-03-26T21:53:46.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:46 smithi055 ceph-mon[29406]: 3.6 deep-scrub ok 2024-03-26T21:53:46.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:46 smithi055 ceph-mon[40587]: 3.6 deep-scrub starts 2024-03-26T21:53:46.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:46 smithi055 ceph-mon[40587]: 3.6 deep-scrub ok 2024-03-26T21:53:46.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:46 smithi067 ceph-mon[33346]: 3.6 deep-scrub starts 2024-03-26T21:53:46.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:46 smithi067 ceph-mon[33346]: 3.6 deep-scrub ok 2024-03-26T21:53:47.425 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:47 smithi055 ceph-mon[29406]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:47.425 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:47 smithi055 ceph-mon[40587]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:47.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:47 smithi067 ceph-mon[33346]: pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:48.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:48.254 INFO:teuthology.orchestra.run.smithi055.stdout:571230650619 2024-03-26T21:53:48.539 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:48 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/59780115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:53:48.539 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:48 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:48.539 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:48 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/59780115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:53:48.540 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:48 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:48.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:48 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/59780115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T21:53:48.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:48 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:53:49.067 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:48 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:49.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[40587]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:49.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:49.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[40587]: 5.12 scrub starts 2024-03-26T21:53:49.379 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[40587]: 5.12 scrub ok 2024-03-26T21:53:49.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[29406]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:49.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:49.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[29406]: 5.12 scrub starts 2024-03-26T21:53:49.380 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:49 smithi055 ceph-mon[29406]: 5.12 scrub ok 2024-03-26T21:53:49.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:49 smithi067 ceph-mon[33346]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T21:53:49.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:49 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:49.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:49 smithi067 ceph-mon[33346]: 5.12 scrub starts 2024-03-26T21:53:49.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:49 smithi067 ceph-mon[33346]: 5.12 scrub ok 2024-03-26T21:53:50.487 INFO:tasks.cephadm.ceph_manager.ceph:need seq 571230650617 got 571230650619 for osd.4 2024-03-26T21:53:50.487 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:50.654 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:50 smithi055 ceph-mon[40587]: 3.4 scrub starts 2024-03-26T21:53:50.655 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:50 smithi055 ceph-mon[40587]: 3.4 scrub ok 2024-03-26T21:53:50.655 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:50 smithi055 ceph-mon[29406]: 3.4 scrub starts 2024-03-26T21:53:50.655 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:50 smithi055 ceph-mon[29406]: 3.4 scrub ok 2024-03-26T21:53:50.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:50 smithi067 ceph-mon[33346]: 3.4 scrub starts 2024-03-26T21:53:50.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:50 smithi067 ceph-mon[33346]: 3.4 scrub ok 2024-03-26T21:53:50.882 INFO:teuthology.orchestra.run.smithi055.stdout:55834575188 2024-03-26T21:53:51.533 INFO:teuthology.orchestra.run.smithi055.stdout:77309411662 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: 3.f scrub starts 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: 3.f scrub ok 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1427664270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: 4.17 scrub starts 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[40587]: 4.17 scrub ok 2024-03-26T21:53:51.534 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:51.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: 3.f scrub starts 2024-03-26T21:53:51.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: 3.f scrub ok 2024-03-26T21:53:51.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1427664270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:53:51.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: 4.17 scrub starts 2024-03-26T21:53:51.535 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:51 smithi055 ceph-mon[29406]: 4.17 scrub ok 2024-03-26T21:53:51.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T21:53:51.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: 3.f scrub starts 2024-03-26T21:53:51.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: 3.f scrub ok 2024-03-26T21:53:51.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1427664270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T21:53:51.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: 4.17 scrub starts 2024-03-26T21:53:51.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:51 smithi067 ceph-mon[33346]: 4.17 scrub ok 2024-03-26T21:53:52.356 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411658 got 77309411662 for osd.2 2024-03-26T21:53:52.356 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:52.592 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[40587]: 5.a deep-scrub starts 2024-03-26T21:53:52.592 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[40587]: 5.a deep-scrub ok 2024-03-26T21:53:52.592 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/4187232821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:53:52.592 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[40587]: 6.d scrub starts 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[40587]: 6.d scrub ok 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[29406]: 5.a deep-scrub starts 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[29406]: 5.a deep-scrub ok 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/4187232821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[29406]: 6.d scrub starts 2024-03-26T21:53:52.593 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:52 smithi055 ceph-mon[29406]: 6.d scrub ok 2024-03-26T21:53:52.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:52 smithi067 ceph-mon[33346]: 5.a deep-scrub starts 2024-03-26T21:53:52.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:52 smithi067 ceph-mon[33346]: 5.a deep-scrub ok 2024-03-26T21:53:52.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:52 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/4187232821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T21:53:52.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:52 smithi067 ceph-mon[33346]: 6.d scrub starts 2024-03-26T21:53:52.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:52 smithi067 ceph-mon[33346]: 6.d scrub ok 2024-03-26T21:53:53.002 INFO:teuthology.orchestra.run.smithi055.stdout:111669150024 2024-03-26T21:53:53.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:53 smithi055 ceph-mon[40587]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:53.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:53 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1751933115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:53:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:53 smithi055 ceph-mon[29406]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:53.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:53 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1751933115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:53:53.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:53 smithi067 ceph-mon[33346]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T21:53:53.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:53 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1751933115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T21:53:53.745 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575184 got 55834575188 for osd.1 2024-03-26T21:53:53.745 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:54.606 INFO:teuthology.orchestra.run.smithi055.stdout:1185410973848 2024-03-26T21:53:54.607 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:54 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:54.607 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:54 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:54.607 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:54 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:54.607 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:54 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:54.695 INFO:teuthology.orchestra.run.smithi055.stdout:34359738713 2024-03-26T21:53:54.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:54 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:54.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:54 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:55.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:55.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:53:55.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/57778859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:53:55.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: 3.3 scrub starts 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2455947517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[40587]: 3.3 scrub ok 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/57778859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: 3.3 scrub starts 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2455947517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:53:55.568 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:55 smithi055 ceph-mon[29406]: 3.3 scrub ok 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/57778859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: 3.3 scrub starts 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2455947517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T21:53:55.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:55 smithi067 ceph-mon[33346]: 3.3 scrub ok 2024-03-26T21:53:56.130 INFO:teuthology.orchestra.run.smithi055.stdout:176093659454 2024-03-26T21:53:56.484 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:56.484 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:56.484 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: 6.1d scrub starts 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: 6.1d scrub ok 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: 3.1a scrub starts 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: 3.1a scrub ok 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1763860236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: 6.1d scrub starts 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: 6.1d scrub ok 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: 3.1a scrub starts 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: 3.1a scrub ok 2024-03-26T21:53:56.485 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:56 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1763860236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:53:56.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:56.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:56.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T21:53:56.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: 6.1d scrub starts 2024-03-26T21:53:56.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: 6.1d scrub ok 2024-03-26T21:53:56.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: 3.1a scrub starts 2024-03-26T21:53:56.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: 3.1a scrub ok 2024-03-26T21:53:56.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:56 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1763860236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T21:53:56.893 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659451 got 176093659454 for osd.5 2024-03-26T21:53:56.893 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:57.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738710 got 34359738713 for osd.0 2024-03-26T21:53:57.441 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:57.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:57 smithi067 ceph-mon[33346]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:57 smithi055 ceph-mon[40587]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:57 smithi055 ceph-mon[29406]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:53:57.854 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150022 got 111669150024 for osd.3 2024-03-26T21:53:57.854 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:58.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:53:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:53:58.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1185410973843 got 1185410973848 for osd.7 2024-03-26T21:53:58.343 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[29406]: 5.11 scrub starts 2024-03-26T21:53:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[29406]: 5.11 scrub ok 2024-03-26T21:53:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[29406]: 5.1b scrub starts 2024-03-26T21:53:58.567 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[29406]: 5.1b scrub ok 2024-03-26T21:53:58.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[40587]: 5.11 scrub starts 2024-03-26T21:53:58.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[40587]: 5.11 scrub ok 2024-03-26T21:53:58.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[40587]: 5.1b scrub starts 2024-03-26T21:53:58.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-mon[40587]: 5.1b scrub ok 2024-03-26T21:53:58.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:58 smithi067 ceph-mon[33346]: 5.11 scrub starts 2024-03-26T21:53:58.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:58 smithi067 ceph-mon[33346]: 5.11 scrub ok 2024-03-26T21:53:58.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:58 smithi067 ceph-mon[33346]: 5.1b scrub starts 2024-03-26T21:53:58.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:58 smithi067 ceph-mon[33346]: 5.1b scrub ok 2024-03-26T21:53:59.023 INFO:teuthology.orchestra.run.smithi055.stdout:214748365116 2024-03-26T21:53:59.317 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:53:58 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:53:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:53:59.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: purged_snaps scrub starts 2024-03-26T21:53:59.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: purged_snaps scrub ok 2024-03-26T21:53:59.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: 3.8 deep-scrub starts 2024-03-26T21:53:59.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: 3.8 deep-scrub ok 2024-03-26T21:53:59.742 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:53:59 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/2673205804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: purged_snaps scrub starts 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: purged_snaps scrub ok 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: 3.8 deep-scrub starts 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: 3.8 deep-scrub ok 2024-03-26T21:53:59.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/2673205804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: purged_snaps scrub starts 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: purged_snaps scrub ok 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: 3.8 deep-scrub starts 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: 3.8 deep-scrub ok 2024-03-26T21:53:59.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:53:59 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/2673205804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T21:53:59.859 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365111 got 214748365116 for osd.6 2024-03-26T21:53:59.859 DEBUG:teuthology.parallel:result is None 2024-03-26T21:53:59.859 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T21:53:59.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:54:00.109 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 4.3 scrub starts 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 4.3 scrub ok 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 6.2 scrub starts 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 6.2 scrub ok 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 4.4 scrub starts 2024-03-26T21:54:00.496 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[40587]: 4.4 scrub ok 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 4.3 scrub starts 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 4.3 scrub ok 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 6.2 scrub starts 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 6.2 scrub ok 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 4.4 scrub starts 2024-03-26T21:54:00.497 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:00 smithi055 ceph-mon[29406]: 4.4 scrub ok 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 4.3 scrub starts 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 4.3 scrub ok 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 6.2 scrub starts 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 6.2 scrub ok 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 4.4 scrub starts 2024-03-26T21:54:00.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:00 smithi067 ceph-mon[33346]: 4.4 scrub ok 2024-03-26T21:54:01.682 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:01 smithi055 ceph-mon[40587]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-26T21:54:01.682 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:01 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:01.683 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:01 smithi055 ceph-mon[29406]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-26T21:54:01.683 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:01 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:01.709 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:01 smithi067 ceph-mon[33346]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-26T21:54:01.709 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:01 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:02.140 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:54:02.143 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:54:02.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:02 smithi067 ceph-mon[33346]: 3.1c scrub starts 2024-03-26T21:54:02.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:02 smithi067 ceph-mon[33346]: 3.1c scrub ok 2024-03-26T21:54:02.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:02 smithi067 ceph-mon[33346]: 4.10 scrub starts 2024-03-26T21:54:02.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:02 smithi067 ceph-mon[33346]: 4.10 scrub ok 2024-03-26T21:54:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[40587]: 3.1c scrub starts 2024-03-26T21:54:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[40587]: 3.1c scrub ok 2024-03-26T21:54:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[40587]: 4.10 scrub starts 2024-03-26T21:54:02.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[40587]: 4.10 scrub ok 2024-03-26T21:54:02.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[29406]: 3.1c scrub starts 2024-03-26T21:54:02.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[29406]: 3.1c scrub ok 2024-03-26T21:54:02.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[29406]: 4.10 scrub starts 2024-03-26T21:54:02.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:02 smithi055 ceph-mon[29406]: 4.10 scrub ok 2024-03-26T21:54:02.905 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1205,"stamp":"2024-03-26T21:54:01.837705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3146,"num_read_kb":2994,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1780184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1349,"ondisk_log_size":1349,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6519748,"kb_used_data":7604,"kb_used_omap":20,"kb_used_meta":6511659,"kb_avail":743441468,"statfs":{"total":767960285184,"available":761284063232,"internally_reserved":0,"allocated":7786496,"data_stored":4682478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20884,"internal_metadata":6667939436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.463394"},"pg_stats":[{"pgid":"4.c","version":"456'35","reported_seq":852,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422746+0000","last_change":"2024-03-26T21:53:37.605343+0000","last_active":"2024-03-26T21:53:59.422746+0000","last_peered":"2024-03-26T21:53:59.422746+0000","last_clean":"2024-03-26T21:53:59.422746+0000","last_became_active":"2024-03-26T21:52:17.974507+0000","last_became_peered":"2024-03-26T21:52:17.974507+0000","last_unstale":"2024-03-26T21:53:59.422746+0000","last_undegraded":"2024-03-26T21:53:59.422746+0000","last_fullsized":"2024-03-26T21:53:59.422746+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-26T21:53:37.605172+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:53:37.605172+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.319108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":390,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643544+0000","last_change":"2024-03-26T21:53:12.156545+0000","last_active":"2024-03-26T21:53:58.643544+0000","last_peered":"2024-03-26T21:53:58.643544+0000","last_clean":"2024-03-26T21:53:58.643544+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:53:58.643544+0000","last_undegraded":"2024-03-26T21:53:58.643544+0000","last_fullsized":"2024-03-26T21:53:58.643544+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:12.156464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.156464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.708287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"505'13","reported_seq":121,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375982+0000","last_change":"2024-03-26T21:53:18.684219+0000","last_active":"2024-03-26T21:53:55.375982+0000","last_peered":"2024-03-26T21:53:55.375982+0000","last_clean":"2024-03-26T21:53:55.375982+0000","last_became_active":"2024-03-26T21:52:08.293748+0000","last_became_peered":"2024-03-26T21:52:08.293748+0000","last_unstale":"2024-03-26T21:53:55.375982+0000","last_undegraded":"2024-03-26T21:53:55.375982+0000","last_fullsized":"2024-03-26T21:53:55.375982+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'13","last_scrub_stamp":"2024-03-26T21:53:18.684110+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:53:18.684110+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.560249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":751,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423040+0000","last_change":"2024-03-26T21:53:31.694080+0000","last_active":"2024-03-26T21:53:59.423040+0000","last_peered":"2024-03-26T21:53:59.423040+0000","last_clean":"2024-03-26T21:53:59.423040+0000","last_became_active":"2024-03-26T21:52:08.306888+0000","last_became_peered":"2024-03-26T21:52:08.306888+0000","last_unstale":"2024-03-26T21:53:59.423040+0000","last_undegraded":"2024-03-26T21:53:59.423040+0000","last_fullsized":"2024-03-26T21:53:59.423040+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:31.693947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:53:31.693947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:43.995259+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":854,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423081+0000","last_change":"2024-03-26T21:53:17.682610+0000","last_active":"2024-03-26T21:53:59.423081+0000","last_peered":"2024-03-26T21:53:59.423081+0000","last_clean":"2024-03-26T21:53:59.423081+0000","last_became_active":"2024-03-26T21:52:08.306232+0000","last_became_peered":"2024-03-26T21:52:08.306232+0000","last_unstale":"2024-03-26T21:53:59.423081+0000","last_undegraded":"2024-03-26T21:53:59.423081+0000","last_fullsized":"2024-03-26T21:53:59.423081+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-26T21:53:17.682481+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:53:17.682481+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.828789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":948,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393246+0000","last_change":"2024-03-26T21:53:04.375087+0000","last_active":"2024-03-26T21:53:55.393246+0000","last_peered":"2024-03-26T21:53:55.393246+0000","last_clean":"2024-03-26T21:53:55.393246+0000","last_became_active":"2024-03-26T21:53:02.589212+0000","last_became_peered":"2024-03-26T21:53:02.589212+0000","last_unstale":"2024-03-26T21:53:55.393246+0000","last_undegraded":"2024-03-26T21:53:55.393246+0000","last_fullsized":"2024-03-26T21:53:55.393246+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:04.374971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:53:04.374971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:07.489077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1069,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415381+0000","last_change":"2024-03-26T21:53:14.411181+0000","last_active":"2024-03-26T21:53:58.415381+0000","last_peered":"2024-03-26T21:53:58.415381+0000","last_clean":"2024-03-26T21:53:58.415381+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:53:58.415381+0000","last_undegraded":"2024-03-26T21:53:58.415381+0000","last_fullsized":"2024-03-26T21:53:58.415381+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:14.411072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:14.411072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.859704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":95,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.376301+0000","last_change":"2024-03-26T21:53:20.858847+0000","last_active":"2024-03-26T21:53:55.376301+0000","last_peered":"2024-03-26T21:53:55.376301+0000","last_clean":"2024-03-26T21:53:55.376301+0000","last_became_active":"2024-03-26T21:52:08.304158+0000","last_became_peered":"2024-03-26T21:52:08.304158+0000","last_unstale":"2024-03-26T21:53:55.376301+0000","last_undegraded":"2024-03-26T21:53:55.376301+0000","last_fullsized":"2024-03-26T21:53:55.376301+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:20.858690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:09.649543+0000","last_clean_scrub_stamp":"2024-03-26T21:53:20.858690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:43.223047+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":779,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423056+0000","last_change":"2024-03-26T21:53:36.638936+0000","last_active":"2024-03-26T21:53:59.423056+0000","last_peered":"2024-03-26T21:53:59.423056+0000","last_clean":"2024-03-26T21:53:59.423056+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:53:59.423056+0000","last_undegraded":"2024-03-26T21:53:59.423056+0000","last_fullsized":"2024-03-26T21:53:59.423056+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:36.638821+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:53:36.638821+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.913029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":751,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422174+0000","last_change":"2024-03-26T21:52:59.646429+0000","last_active":"2024-03-26T21:53:59.422174+0000","last_peered":"2024-03-26T21:53:59.422174+0000","last_clean":"2024-03-26T21:53:59.422174+0000","last_became_active":"2024-03-26T21:52:08.303111+0000","last_became_peered":"2024-03-26T21:52:08.303111+0000","last_unstale":"2024-03-26T21:53:59.422174+0000","last_undegraded":"2024-03-26T21:53:59.422174+0000","last_fullsized":"2024-03-26T21:53:59.422174+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:59.646315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:52:59.646315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.828998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1057,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393902+0000","last_change":"2024-03-26T21:52:36.728256+0000","last_active":"2024-03-26T21:53:55.393902+0000","last_peered":"2024-03-26T21:53:55.393902+0000","last_clean":"2024-03-26T21:53:55.393902+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:53:55.393902+0000","last_undegraded":"2024-03-26T21:53:55.393902+0000","last_fullsized":"2024-03-26T21:53:55.393902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:36.728175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:36.728175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":382,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643807+0000","last_change":"2024-03-26T21:52:29.072033+0000","last_active":"2024-03-26T21:53:58.643807+0000","last_peered":"2024-03-26T21:53:58.643807+0000","last_clean":"2024-03-26T21:53:58.643807+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:53:58.643807+0000","last_undegraded":"2024-03-26T21:53:58.643807+0000","last_fullsized":"2024-03-26T21:53:58.643807+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:29.071901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:52:29.071901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1206,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415520+0000","last_change":"2024-03-26T21:53:29.289553+0000","last_active":"2024-03-26T21:53:58.415520+0000","last_peered":"2024-03-26T21:53:58.415520+0000","last_clean":"2024-03-26T21:53:58.415520+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:53:58.415520+0000","last_undegraded":"2024-03-26T21:53:58.415520+0000","last_fullsized":"2024-03-26T21:53:58.415520+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:53:29.289458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:29.289458+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.817050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":382,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642563+0000","last_change":"2024-03-26T21:52:20.010834+0000","last_active":"2024-03-26T21:53:58.642563+0000","last_peered":"2024-03-26T21:53:58.642563+0000","last_clean":"2024-03-26T21:53:58.642563+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:53:58.642563+0000","last_undegraded":"2024-03-26T21:53:58.642563+0000","last_fullsized":"2024-03-26T21:53:58.642563+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:20.010747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:43:06.961290+0000","last_clean_scrub_stamp":"2024-03-26T21:52:20.010747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"512'13","reported_seq":958,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423185+0000","last_change":"2024-03-26T21:53:38.608081+0000","last_active":"2024-03-26T21:53:59.423185+0000","last_peered":"2024-03-26T21:53:59.423185+0000","last_clean":"2024-03-26T21:53:59.423185+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:53:59.423185+0000","last_undegraded":"2024-03-26T21:53:59.423185+0000","last_fullsized":"2024-03-26T21:53:59.423185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'13","last_scrub_stamp":"2024-03-26T21:53:38.607961+0000","last_deep_scrub":"512'13","last_deep_scrub_stamp":"2024-03-26T21:52:25.766920+0000","last_clean_scrub_stamp":"2024-03-26T21:53:38.607961+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.096308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1040,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392993+0000","last_change":"2024-03-26T21:53:51.789465+0000","last_active":"2024-03-26T21:53:55.392993+0000","last_peered":"2024-03-26T21:53:55.392993+0000","last_clean":"2024-03-26T21:53:55.392993+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:53:55.392993+0000","last_undegraded":"2024-03-26T21:53:55.392993+0000","last_fullsized":"2024-03-26T21:53:55.392993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:51.789350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:53:51.789350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.358896+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":758,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422396+0000","last_change":"2024-03-26T21:53:34.669331+0000","last_active":"2024-03-26T21:53:59.422396+0000","last_peered":"2024-03-26T21:53:59.422396+0000","last_clean":"2024-03-26T21:53:59.422396+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:53:59.422396+0000","last_undegraded":"2024-03-26T21:53:59.422396+0000","last_fullsized":"2024-03-26T21:53:59.422396+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.669230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:10.787528+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.669230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:35.981381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":426,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642455+0000","last_change":"2024-03-26T21:53:39.184126+0000","last_active":"2024-03-26T21:53:58.642455+0000","last_peered":"2024-03-26T21:53:58.642455+0000","last_clean":"2024-03-26T21:53:58.642455+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:53:58.642455+0000","last_undegraded":"2024-03-26T21:53:58.642455+0000","last_fullsized":"2024-03-26T21:53:58.642455+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'50","last_scrub_stamp":"2024-03-26T21:53:39.183998+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.183998+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:51.122789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":390,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644128+0000","last_change":"2024-03-26T21:53:16.195641+0000","last_active":"2024-03-26T21:53:58.644128+0000","last_peered":"2024-03-26T21:53:58.644128+0000","last_clean":"2024-03-26T21:53:58.644128+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:53:58.644128+0000","last_undegraded":"2024-03-26T21:53:58.644128+0000","last_fullsized":"2024-03-26T21:53:58.644128+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:16.195538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:16.195538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:26.536932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":771,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.666521+0000","last_change":"2024-03-26T21:53:59.666546+0000","last_active":"2024-03-26T21:53:59.666521+0000","last_peered":"2024-03-26T21:53:59.666521+0000","last_clean":"2024-03-26T21:53:59.666521+0000","last_became_active":"2024-03-26T21:52:08.303240+0000","last_became_peered":"2024-03-26T21:52:08.303240+0000","last_unstale":"2024-03-26T21:53:59.666521+0000","last_undegraded":"2024-03-26T21:53:59.666521+0000","last_fullsized":"2024-03-26T21:53:59.666521+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:59.666446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:53:59.666446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:00.275800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":504,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642641+0000","last_change":"2024-03-26T21:53:15.213177+0000","last_active":"2024-03-26T21:53:58.642641+0000","last_peered":"2024-03-26T21:53:58.642641+0000","last_clean":"2024-03-26T21:53:58.642641+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:53:58.642641+0000","last_undegraded":"2024-03-26T21:53:58.642641+0000","last_fullsized":"2024-03-26T21:53:58.642641+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:53:15.213078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.213078+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:20.643458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"534'49","reported_seq":804,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422270+0000","last_change":"2024-03-26T21:53:19.655516+0000","last_active":"2024-03-26T21:53:59.422270+0000","last_peered":"2024-03-26T21:53:59.422270+0000","last_clean":"2024-03-26T21:53:59.422270+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:53:59.422270+0000","last_undegraded":"2024-03-26T21:53:59.422270+0000","last_fullsized":"2024-03-26T21:53:59.422270+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'49","last_scrub_stamp":"2024-03-26T21:53:19.655382+0000","last_deep_scrub":"534'49","last_deep_scrub_stamp":"2024-03-26T21:53:19.655382+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.655382+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:44.067095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"537'2","reported_seq":1055,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643568+0000","last_change":"2024-03-26T21:53:45.324633+0000","last_active":"2024-03-26T21:53:58.643568+0000","last_peered":"2024-03-26T21:53:58.643568+0000","last_clean":"2024-03-26T21:53:58.643568+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:53:58.643568+0000","last_undegraded":"2024-03-26T21:53:58.643568+0000","last_fullsized":"2024-03-26T21:53:58.643568+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'2","last_scrub_stamp":"2024-03-26T21:53:45.324543+0000","last_deep_scrub":"537'2","last_deep_scrub_stamp":"2024-03-26T21:53:45.324543+0000","last_clean_scrub_stamp":"2024-03-26T21:53:45.324543+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:05.741057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":962,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644819+0000","last_change":"2024-03-26T21:52:38.404963+0000","last_active":"2024-03-26T21:53:58.644819+0000","last_peered":"2024-03-26T21:53:58.644819+0000","last_clean":"2024-03-26T21:53:58.644819+0000","last_became_active":"2024-03-26T21:52:08.296363+0000","last_became_peered":"2024-03-26T21:52:08.296363+0000","last_unstale":"2024-03-26T21:53:58.644819+0000","last_undegraded":"2024-03-26T21:53:58.644819+0000","last_fullsized":"2024-03-26T21:53:58.644819+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.404842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.404842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:01.684210+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1346,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643192+0000","last_change":"2024-03-26T21:53:32.373011+0000","last_active":"2024-03-26T21:53:58.643192+0000","last_peered":"2024-03-26T21:53:58.643192+0000","last_clean":"2024-03-26T21:53:58.643192+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:53:58.643192+0000","last_undegraded":"2024-03-26T21:53:58.643192+0000","last_fullsized":"2024-03-26T21:53:58.643192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:53:32.372928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.372928+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.640659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":1135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415044+0000","last_change":"2024-03-26T21:53:35.317121+0000","last_active":"2024-03-26T21:53:58.415044+0000","last_peered":"2024-03-26T21:53:58.415044+0000","last_clean":"2024-03-26T21:53:58.415044+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:53:58.415044+0000","last_undegraded":"2024-03-26T21:53:58.415044+0000","last_fullsized":"2024-03-26T21:53:58.415044+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-26T21:53:35.317003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.317003+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.991721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1077,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393233+0000","last_change":"2024-03-26T21:52:30.766822+0000","last_active":"2024-03-26T21:53:55.393233+0000","last_peered":"2024-03-26T21:53:55.393233+0000","last_clean":"2024-03-26T21:53:55.393233+0000","last_became_active":"2024-03-26T21:52:08.294134+0000","last_became_peered":"2024-03-26T21:52:08.294134+0000","last_unstale":"2024-03-26T21:53:55.393233+0000","last_undegraded":"2024-03-26T21:53:55.393233+0000","last_fullsized":"2024-03-26T21:53:55.393233+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:30.766706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:52:30.766706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:58.557141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1045,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642490+0000","last_change":"2024-03-26T21:52:58.505853+0000","last_active":"2024-03-26T21:53:58.642490+0000","last_peered":"2024-03-26T21:53:58.642490+0000","last_clean":"2024-03-26T21:53:58.642490+0000","last_became_active":"2024-03-26T21:52:08.301864+0000","last_became_peered":"2024-03-26T21:52:08.301864+0000","last_unstale":"2024-03-26T21:53:58.642490+0000","last_undegraded":"2024-03-26T21:53:58.642490+0000","last_fullsized":"2024-03-26T21:53:58.642490+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:58.505725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:58.505725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:19.925855+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":919,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393703+0000","last_change":"2024-03-26T21:53:19.361318+0000","last_active":"2024-03-26T21:53:55.393703+0000","last_peered":"2024-03-26T21:53:55.393703+0000","last_clean":"2024-03-26T21:53:55.393703+0000","last_became_active":"2024-03-26T21:53:02.593067+0000","last_became_peered":"2024-03-26T21:53:02.593067+0000","last_unstale":"2024-03-26T21:53:55.393703+0000","last_undegraded":"2024-03-26T21:53:55.393703+0000","last_fullsized":"2024-03-26T21:53:55.393703+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:19.361198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.361198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.441219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":1104,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643300+0000","last_change":"2024-03-26T21:52:46.390309+0000","last_active":"2024-03-26T21:53:58.643300+0000","last_peered":"2024-03-26T21:53:58.643300+0000","last_clean":"2024-03-26T21:53:58.643300+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:53:58.643300+0000","last_undegraded":"2024-03-26T21:53:58.643300+0000","last_fullsized":"2024-03-26T21:53:58.643300+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:52:46.390213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:52:46.390213+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1077,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415268+0000","last_change":"2024-03-26T21:53:49.329111+0000","last_active":"2024-03-26T21:53:58.415268+0000","last_peered":"2024-03-26T21:53:58.415268+0000","last_clean":"2024-03-26T21:53:58.415268+0000","last_became_active":"2024-03-26T21:52:08.312067+0000","last_became_peered":"2024-03-26T21:52:08.312067+0000","last_unstale":"2024-03-26T21:53:58.415268+0000","last_undegraded":"2024-03-26T21:53:58.415268+0000","last_fullsized":"2024-03-26T21:53:58.415268+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:49.329008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:53:49.329008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.054935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1049,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.414593+0000","last_change":"2024-03-26T21:52:38.375617+0000","last_active":"2024-03-26T21:53:58.414593+0000","last_peered":"2024-03-26T21:53:58.414593+0000","last_clean":"2024-03-26T21:53:58.414593+0000","last_became_active":"2024-03-26T21:52:08.312506+0000","last_became_peered":"2024-03-26T21:52:08.312506+0000","last_unstale":"2024-03-26T21:53:58.414593+0000","last_undegraded":"2024-03-26T21:53:58.414593+0000","last_fullsized":"2024-03-26T21:53:58.414593+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.375489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:38.375489+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.375489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:07.708381+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1044,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393889+0000","last_change":"2024-03-26T21:53:38.811091+0000","last_active":"2024-03-26T21:53:55.393889+0000","last_peered":"2024-03-26T21:53:55.393889+0000","last_clean":"2024-03-26T21:53:55.393889+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:53:55.393889+0000","last_undegraded":"2024-03-26T21:53:55.393889+0000","last_fullsized":"2024-03-26T21:53:55.393889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:38.810965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:38.810965+0000","last_clean_scrub_stamp":"2024-03-26T21:53:38.810965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:45.765995+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":1150,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643547+0000","last_change":"2024-03-26T21:53:41.307161+0000","last_active":"2024-03-26T21:53:58.643547+0000","last_peered":"2024-03-26T21:53:58.643547+0000","last_clean":"2024-03-26T21:53:58.643547+0000","last_became_active":"2024-03-26T21:52:08.299332+0000","last_became_peered":"2024-03-26T21:52:08.299332+0000","last_unstale":"2024-03-26T21:53:58.643547+0000","last_undegraded":"2024-03-26T21:53:58.643547+0000","last_fullsized":"2024-03-26T21:53:58.643547+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'36","last_scrub_stamp":"2024-03-26T21:53:41.307050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.307050+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:55.228833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2415,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642925+0000","last_change":"2024-03-26T21:53:32.519457+0000","last_active":"2024-03-26T21:53:58.642925+0000","last_peered":"2024-03-26T21:53:58.642925+0000","last_clean":"2024-03-26T21:53:58.642925+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:53:58.642925+0000","last_undegraded":"2024-03-26T21:53:58.642925+0000","last_fullsized":"2024-03-26T21:53:58.642925+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:53:32.519338+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:53:32.519338+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.519338+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.021475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1402,"num_read_kb":1397,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1066,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643567+0000","last_change":"2024-03-26T21:52:54.317852+0000","last_active":"2024-03-26T21:53:58.643567+0000","last_peered":"2024-03-26T21:53:58.643567+0000","last_clean":"2024-03-26T21:53:58.643567+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:53:58.643567+0000","last_undegraded":"2024-03-26T21:53:58.643567+0000","last_fullsized":"2024-03-26T21:53:58.643567+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:54.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:52:54.317687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:02.719799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1057,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.414975+0000","last_change":"2024-03-26T21:53:12.442195+0000","last_active":"2024-03-26T21:53:58.414975+0000","last_peered":"2024-03-26T21:53:58.414975+0000","last_clean":"2024-03-26T21:53:58.414975+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:53:58.414975+0000","last_undegraded":"2024-03-26T21:53:58.414975+0000","last_fullsized":"2024-03-26T21:53:58.414975+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:12.442108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.442108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.535711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":987,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644546+0000","last_change":"2024-03-26T21:53:39.555788+0000","last_active":"2024-03-26T21:53:58.644546+0000","last_peered":"2024-03-26T21:53:58.644546+0000","last_clean":"2024-03-26T21:53:58.644546+0000","last_became_active":"2024-03-26T21:52:08.295236+0000","last_became_peered":"2024-03-26T21:52:08.295236+0000","last_unstale":"2024-03-26T21:53:58.644546+0000","last_undegraded":"2024-03-26T21:53:58.644546+0000","last_fullsized":"2024-03-26T21:53:58.644546+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:39.555685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.555685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.805657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"531'56","reported_seq":1315,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393342+0000","last_change":"2024-03-26T21:53:02.594215+0000","last_active":"2024-03-26T21:53:55.393342+0000","last_peered":"2024-03-26T21:53:55.393342+0000","last_clean":"2024-03-26T21:53:55.393342+0000","last_became_active":"2024-03-26T21:53:02.593742+0000","last_became_peered":"2024-03-26T21:53:02.593742+0000","last_unstale":"2024-03-26T21:53:55.393342+0000","last_undegraded":"2024-03-26T21:53:55.393342+0000","last_fullsized":"2024-03-26T21:53:55.393342+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'55","last_scrub_stamp":"2024-03-26T21:52:45.761346+0000","last_deep_scrub":"468'55","last_deep_scrub_stamp":"2024-03-26T21:52:45.761346+0000","last_clean_scrub_stamp":"2024-03-26T21:52:45.761346+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.939882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":390,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643093+0000","last_change":"2024-03-26T21:53:33.185272+0000","last_active":"2024-03-26T21:53:58.643093+0000","last_peered":"2024-03-26T21:53:58.643093+0000","last_clean":"2024-03-26T21:53:58.643093+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:53:58.643093+0000","last_undegraded":"2024-03-26T21:53:58.643093+0000","last_fullsized":"2024-03-26T21:53:58.643093+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:33.185189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:53:33.185189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.817991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1236,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644499+0000","last_change":"2024-03-26T21:52:52.366694+0000","last_active":"2024-03-26T21:53:58.644499+0000","last_peered":"2024-03-26T21:53:58.644499+0000","last_clean":"2024-03-26T21:53:58.644499+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:53:58.644499+0000","last_undegraded":"2024-03-26T21:53:58.644499+0000","last_fullsized":"2024-03-26T21:53:58.644499+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:52:52.366588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:52:52.366588+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.435534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1065,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393737+0000","last_change":"2024-03-26T21:52:38.814451+0000","last_active":"2024-03-26T21:53:55.393737+0000","last_peered":"2024-03-26T21:53:55.393737+0000","last_clean":"2024-03-26T21:53:55.393737+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:53:55.393737+0000","last_undegraded":"2024-03-26T21:53:55.393737+0000","last_fullsized":"2024-03-26T21:53:55.393737+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.814353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.814353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":95,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.376169+0000","last_change":"2024-03-26T21:53:29.782827+0000","last_active":"2024-03-26T21:53:55.376169+0000","last_peered":"2024-03-26T21:53:55.376169+0000","last_clean":"2024-03-26T21:53:55.376169+0000","last_became_active":"2024-03-26T21:52:08.296077+0000","last_became_peered":"2024-03-26T21:52:08.296077+0000","last_unstale":"2024-03-26T21:53:55.376169+0000","last_undegraded":"2024-03-26T21:53:55.376169+0000","last_fullsized":"2024-03-26T21:53:55.376169+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:29.782734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:53:29.782734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.219050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1096,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415651+0000","last_change":"2024-03-26T21:52:42.368231+0000","last_active":"2024-03-26T21:53:58.415651+0000","last_peered":"2024-03-26T21:53:58.415651+0000","last_clean":"2024-03-26T21:53:58.415651+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:53:58.415651+0000","last_undegraded":"2024-03-26T21:53:58.415651+0000","last_fullsized":"2024-03-26T21:53:58.415651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-26T21:52:42.368124+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:52:42.368124+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:05.954104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375522+0000","last_change":"2024-03-26T21:53:34.784060+0000","last_active":"2024-03-26T21:53:55.375522+0000","last_peered":"2024-03-26T21:53:55.375522+0000","last_clean":"2024-03-26T21:53:55.375522+0000","last_became_active":"2024-03-26T21:52:08.303990+0000","last_became_peered":"2024-03-26T21:52:08.303990+0000","last_unstale":"2024-03-26T21:53:55.375522+0000","last_undegraded":"2024-03-26T21:53:55.375522+0000","last_fullsized":"2024-03-26T21:53:55.375522+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.783951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:34.783951+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.783951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:37.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1077,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.414749+0000","last_change":"2024-03-26T21:53:07.427049+0000","last_active":"2024-03-26T21:53:58.414749+0000","last_peered":"2024-03-26T21:53:58.414749+0000","last_clean":"2024-03-26T21:53:58.414749+0000","last_became_active":"2024-03-26T21:52:08.311751+0000","last_became_peered":"2024-03-26T21:52:08.311751+0000","last_unstale":"2024-03-26T21:53:58.414749+0000","last_undegraded":"2024-03-26T21:53:58.414749+0000","last_fullsized":"2024-03-26T21:53:58.414749+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:07.426936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:53:07.426936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.805222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":955,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643317+0000","last_change":"2024-03-26T21:53:01.495558+0000","last_active":"2024-03-26T21:53:58.643317+0000","last_peered":"2024-03-26T21:53:58.643317+0000","last_clean":"2024-03-26T21:53:58.643317+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:53:58.643317+0000","last_undegraded":"2024-03-26T21:53:58.643317+0000","last_fullsized":"2024-03-26T21:53:58.643317+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.495451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.495451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.142605+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642915+0000","last_change":"2024-03-26T21:52:36.359809+0000","last_active":"2024-03-26T21:53:58.642915+0000","last_peered":"2024-03-26T21:53:58.642915+0000","last_clean":"2024-03-26T21:53:58.642915+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:53:58.642915+0000","last_undegraded":"2024-03-26T21:53:58.642915+0000","last_fullsized":"2024-03-26T21:53:58.642915+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:36.359706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:52:36.359706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:58.945690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'67","reported_seq":1507,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392840+0000","last_change":"2024-03-26T21:52:28.733506+0000","last_active":"2024-03-26T21:53:55.392840+0000","last_peered":"2024-03-26T21:53:55.392840+0000","last_clean":"2024-03-26T21:53:55.392840+0000","last_became_active":"2024-03-26T21:52:08.299794+0000","last_became_peered":"2024-03-26T21:52:08.299794+0000","last_unstale":"2024-03-26T21:53:55.392840+0000","last_undegraded":"2024-03-26T21:53:55.392840+0000","last_fullsized":"2024-03-26T21:53:55.392840+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'67","last_scrub_stamp":"2024-03-26T21:52:28.733406+0000","last_deep_scrub":"513'67","last_deep_scrub_stamp":"2024-03-26T21:52:28.733406+0000","last_clean_scrub_stamp":"2024-03-26T21:52:28.733406+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422223+0000","last_change":"2024-03-26T21:53:54.702335+0000","last_active":"2024-03-26T21:53:59.422223+0000","last_peered":"2024-03-26T21:53:59.422223+0000","last_clean":"2024-03-26T21:53:59.422223+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:53:59.422223+0000","last_undegraded":"2024-03-26T21:53:59.422223+0000","last_fullsized":"2024-03-26T21:53:59.422223+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:54.702226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:41.701840+0000","last_clean_scrub_stamp":"2024-03-26T21:53:54.702226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:21.217308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":1063,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392654+0000","last_change":"2024-03-26T21:52:32.770619+0000","last_active":"2024-03-26T21:53:55.392654+0000","last_peered":"2024-03-26T21:53:55.392654+0000","last_clean":"2024-03-26T21:53:55.392654+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:53:55.392654+0000","last_undegraded":"2024-03-26T21:53:55.392654+0000","last_fullsized":"2024-03-26T21:53:55.392654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:52:32.770524+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:52:32.770524+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":382,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643985+0000","last_change":"2024-03-26T21:52:33.083793+0000","last_active":"2024-03-26T21:53:58.643985+0000","last_peered":"2024-03-26T21:53:58.643985+0000","last_clean":"2024-03-26T21:53:58.643985+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:53:58.643985+0000","last_undegraded":"2024-03-26T21:53:58.643985+0000","last_fullsized":"2024-03-26T21:53:58.643985+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:33.083710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:33.083710+0000","last_clean_scrub_stamp":"2024-03-26T21:52:33.083710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1111,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643422+0000","last_change":"2024-03-26T21:53:21.491076+0000","last_active":"2024-03-26T21:53:58.643422+0000","last_peered":"2024-03-26T21:53:58.643422+0000","last_clean":"2024-03-26T21:53:58.643422+0000","last_became_active":"2024-03-26T21:52:17.976074+0000","last_became_peered":"2024-03-26T21:52:17.976074+0000","last_unstale":"2024-03-26T21:53:58.643422+0000","last_undegraded":"2024-03-26T21:53:58.643422+0000","last_fullsized":"2024-03-26T21:53:58.643422+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-26T21:53:21.490925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:21.490925+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:36.818825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1072,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393411+0000","last_change":"2024-03-26T21:53:11.707647+0000","last_active":"2024-03-26T21:53:55.393411+0000","last_peered":"2024-03-26T21:53:55.393411+0000","last_clean":"2024-03-26T21:53:55.393411+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:53:55.393411+0000","last_undegraded":"2024-03-26T21:53:55.393411+0000","last_fullsized":"2024-03-26T21:53:55.393411+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:11.707547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:53:11.707547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.254548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":95,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375811+0000","last_change":"2024-03-26T21:53:50.795495+0000","last_active":"2024-03-26T21:53:55.375811+0000","last_peered":"2024-03-26T21:53:55.375811+0000","last_clean":"2024-03-26T21:53:55.375811+0000","last_became_active":"2024-03-26T21:52:08.297815+0000","last_became_peered":"2024-03-26T21:52:08.297815+0000","last_unstale":"2024-03-26T21:53:55.375811+0000","last_undegraded":"2024-03-26T21:53:55.375811+0000","last_fullsized":"2024-03-26T21:53:55.375811+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:50.795415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:50.795415+0000","last_clean_scrub_stamp":"2024-03-26T21:53:50.795415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:15.674635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1041,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643708+0000","last_change":"2024-03-26T21:53:30.480975+0000","last_active":"2024-03-26T21:53:58.643708+0000","last_peered":"2024-03-26T21:53:58.643708+0000","last_clean":"2024-03-26T21:53:58.643708+0000","last_became_active":"2024-03-26T21:52:08.299836+0000","last_became_peered":"2024-03-26T21:52:08.299836+0000","last_unstale":"2024-03-26T21:53:58.643708+0000","last_undegraded":"2024-03-26T21:53:58.643708+0000","last_fullsized":"2024-03-26T21:53:58.643708+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:30.480878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.480878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.420091+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1020,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393593+0000","last_change":"2024-03-26T21:53:26.261237+0000","last_active":"2024-03-26T21:53:55.393593+0000","last_peered":"2024-03-26T21:53:55.393593+0000","last_clean":"2024-03-26T21:53:55.393593+0000","last_became_active":"2024-03-26T21:53:02.592322+0000","last_became_peered":"2024-03-26T21:53:02.592322+0000","last_unstale":"2024-03-26T21:53:55.393593+0000","last_undegraded":"2024-03-26T21:53:55.393593+0000","last_fullsized":"2024-03-26T21:53:55.393593+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'60","last_scrub_stamp":"2024-03-26T21:53:26.261097+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:53:26.261097+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.939070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":989,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643739+0000","last_change":"2024-03-26T21:52:42.431373+0000","last_active":"2024-03-26T21:53:58.643739+0000","last_peered":"2024-03-26T21:53:58.643739+0000","last_clean":"2024-03-26T21:53:58.643739+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:53:58.643739+0000","last_undegraded":"2024-03-26T21:53:58.643739+0000","last_fullsized":"2024-03-26T21:53:58.643739+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:42.431264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:44.612470+0000","last_clean_scrub_stamp":"2024-03-26T21:52:42.431264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:01.611698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":87,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375485+0000","last_change":"2024-03-26T21:53:16.778514+0000","last_active":"2024-03-26T21:53:55.375485+0000","last_peered":"2024-03-26T21:53:55.375485+0000","last_clean":"2024-03-26T21:53:55.375485+0000","last_became_active":"2024-03-26T21:52:08.303310+0000","last_became_peered":"2024-03-26T21:52:08.303310+0000","last_unstale":"2024-03-26T21:53:55.375485+0000","last_undegraded":"2024-03-26T21:53:55.375485+0000","last_fullsized":"2024-03-26T21:53:55.375485+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:16.778380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:02.460651+0000","last_clean_scrub_stamp":"2024-03-26T21:53:16.778380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.517410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":974,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642882+0000","last_change":"2024-03-26T21:53:13.668492+0000","last_active":"2024-03-26T21:53:58.642882+0000","last_peered":"2024-03-26T21:53:58.642882+0000","last_clean":"2024-03-26T21:53:58.642882+0000","last_became_active":"2024-03-26T21:52:08.299894+0000","last_became_peered":"2024-03-26T21:52:08.299894+0000","last_unstale":"2024-03-26T21:53:58.642882+0000","last_undegraded":"2024-03-26T21:53:58.642882+0000","last_fullsized":"2024-03-26T21:53:58.642882+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:13.668408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:13.668408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:38.983061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375117+0000","last_change":"2024-03-26T21:53:24.771160+0000","last_active":"2024-03-26T21:53:55.375117+0000","last_peered":"2024-03-26T21:53:55.375117+0000","last_clean":"2024-03-26T21:53:55.375117+0000","last_became_active":"2024-03-26T21:52:08.304374+0000","last_became_peered":"2024-03-26T21:52:08.304374+0000","last_unstale":"2024-03-26T21:53:55.375117+0000","last_undegraded":"2024-03-26T21:53:55.375117+0000","last_fullsized":"2024-03-26T21:53:55.375117+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.771068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.771068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.161836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":819,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422324+0000","last_change":"2024-03-26T21:53:10.746777+0000","last_active":"2024-03-26T21:53:59.422324+0000","last_peered":"2024-03-26T21:53:59.422324+0000","last_clean":"2024-03-26T21:53:59.422324+0000","last_became_active":"2024-03-26T21:52:17.977990+0000","last_became_peered":"2024-03-26T21:52:17.977990+0000","last_unstale":"2024-03-26T21:53:59.422324+0000","last_undegraded":"2024-03-26T21:53:59.422324+0000","last_fullsized":"2024-03-26T21:53:59.422324+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-26T21:53:10.746662+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:53:10.746662+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:24.942562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":1019,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644000+0000","last_change":"2024-03-26T21:52:48.378422+0000","last_active":"2024-03-26T21:53:58.644000+0000","last_peered":"2024-03-26T21:53:58.644000+0000","last_clean":"2024-03-26T21:53:58.644000+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:53:58.644000+0000","last_undegraded":"2024-03-26T21:53:58.644000+0000","last_fullsized":"2024-03-26T21:53:58.644000+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:52:48.378250+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:52:48.378250+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:04.429955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":390,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643448+0000","last_change":"2024-03-26T21:53:34.344239+0000","last_active":"2024-03-26T21:53:58.643448+0000","last_peered":"2024-03-26T21:53:58.643448+0000","last_clean":"2024-03-26T21:53:58.643448+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:53:58.643448+0000","last_undegraded":"2024-03-26T21:53:58.643448+0000","last_fullsized":"2024-03-26T21:53:58.643448+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.344127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:34.344127+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.344127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:59.284054+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"534'15","reported_seq":1288,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643464+0000","last_change":"2024-03-26T21:53:15.609556+0000","last_active":"2024-03-26T21:53:58.643464+0000","last_peered":"2024-03-26T21:53:58.643464+0000","last_clean":"2024-03-26T21:53:58.643464+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:53:58.643464+0000","last_undegraded":"2024-03-26T21:53:58.643464+0000","last_fullsized":"2024-03-26T21:53:58.643464+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'15","last_scrub_stamp":"2024-03-26T21:53:15.609420+0000","last_deep_scrub":"534'15","last_deep_scrub_stamp":"2024-03-26T21:53:15.609420+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.609420+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.962929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":1125,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392539+0000","last_change":"2024-03-26T21:53:06.764246+0000","last_active":"2024-03-26T21:53:55.392539+0000","last_peered":"2024-03-26T21:53:55.392539+0000","last_clean":"2024-03-26T21:53:55.392539+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:53:55.392539+0000","last_undegraded":"2024-03-26T21:53:55.392539+0000","last_fullsized":"2024-03-26T21:53:55.392539+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:53:06.764090+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:53:06.764090+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.496953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":1036,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643634+0000","last_change":"2024-03-26T21:53:50.604486+0000","last_active":"2024-03-26T21:53:58.643634+0000","last_peered":"2024-03-26T21:53:58.643634+0000","last_clean":"2024-03-26T21:53:58.643634+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:53:58.643634+0000","last_undegraded":"2024-03-26T21:53:58.643634+0000","last_fullsized":"2024-03-26T21:53:58.643634+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:53:50.604363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:50.604363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.940740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1040,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393017+0000","last_change":"2024-03-26T21:53:07.773451+0000","last_active":"2024-03-26T21:53:55.393017+0000","last_peered":"2024-03-26T21:53:55.393017+0000","last_clean":"2024-03-26T21:53:55.393017+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:53:55.393017+0000","last_undegraded":"2024-03-26T21:53:55.393017+0000","last_fullsized":"2024-03-26T21:53:55.393017+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:07.773327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:07.773327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:08.760526+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":420,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644019+0000","last_change":"2024-03-26T21:53:51.090602+0000","last_active":"2024-03-26T21:53:58.644019+0000","last_peered":"2024-03-26T21:53:58.644019+0000","last_clean":"2024-03-26T21:53:58.644019+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:53:58.644019+0000","last_undegraded":"2024-03-26T21:53:58.644019+0000","last_fullsized":"2024-03-26T21:53:58.644019+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'18","last_scrub_stamp":"2024-03-26T21:53:51.090514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:51.090514+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.695920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":926,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392633+0000","last_change":"2024-03-26T21:53:02.594560+0000","last_active":"2024-03-26T21:53:55.392633+0000","last_peered":"2024-03-26T21:53:55.392633+0000","last_clean":"2024-03-26T21:53:55.392633+0000","last_became_active":"2024-03-26T21:53:02.594195+0000","last_became_peered":"2024-03-26T21:53:02.594195+0000","last_unstale":"2024-03-26T21:53:55.392633+0000","last_undegraded":"2024-03-26T21:53:55.392633+0000","last_fullsized":"2024-03-26T21:53:55.392633+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.380797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.380797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:21.247952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1055,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393262+0000","last_change":"2024-03-26T21:52:50.710856+0000","last_active":"2024-03-26T21:53:55.393262+0000","last_peered":"2024-03-26T21:53:55.393262+0000","last_clean":"2024-03-26T21:53:55.393262+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:53:55.393262+0000","last_undegraded":"2024-03-26T21:53:55.393262+0000","last_fullsized":"2024-03-26T21:53:55.393262+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:50.710777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.710777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:59.799184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":972,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643428+0000","last_change":"2024-03-26T21:53:35.611244+0000","last_active":"2024-03-26T21:53:58.643428+0000","last_peered":"2024-03-26T21:53:58.643428+0000","last_clean":"2024-03-26T21:53:58.643428+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:53:58.643428+0000","last_undegraded":"2024-03-26T21:53:58.643428+0000","last_fullsized":"2024-03-26T21:53:58.643428+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.611124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:22.364735+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.611124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:39.399290+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1104,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643674+0000","last_change":"2024-03-26T21:53:04.374991+0000","last_active":"2024-03-26T21:53:58.643674+0000","last_peered":"2024-03-26T21:53:58.643674+0000","last_clean":"2024-03-26T21:53:58.643674+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:53:58.643674+0000","last_undegraded":"2024-03-26T21:53:58.643674+0000","last_fullsized":"2024-03-26T21:53:58.643674+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:04.374896+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:53:04.374896+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.011033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1009,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643717+0000","last_change":"2024-03-26T21:53:14.653735+0000","last_active":"2024-03-26T21:53:58.643717+0000","last_peered":"2024-03-26T21:53:58.643717+0000","last_clean":"2024-03-26T21:53:58.643717+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:53:58.643717+0000","last_undegraded":"2024-03-26T21:53:58.643717+0000","last_fullsized":"2024-03-26T21:53:58.643717+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:14.653627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:14.653627+0000","last_clean_scrub_stamp":"2024-03-26T21:53:14.653627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:32.483533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":700,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415563+0000","last_change":"2024-03-26T21:53:15.374854+0000","last_active":"2024-03-26T21:53:58.415563+0000","last_peered":"2024-03-26T21:53:58.415563+0000","last_clean":"2024-03-26T21:53:58.415563+0000","last_became_active":"2024-03-26T21:52:08.312931+0000","last_became_peered":"2024-03-26T21:52:08.312931+0000","last_unstale":"2024-03-26T21:53:58.415563+0000","last_undegraded":"2024-03-26T21:53:58.415563+0000","last_fullsized":"2024-03-26T21:53:58.415563+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:15.374770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.374770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:20.037797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":87,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375908+0000","last_change":"2024-03-26T21:53:03.653511+0000","last_active":"2024-03-26T21:53:55.375908+0000","last_peered":"2024-03-26T21:53:55.375908+0000","last_clean":"2024-03-26T21:53:55.375908+0000","last_became_active":"2024-03-26T21:52:08.295874+0000","last_became_peered":"2024-03-26T21:52:08.295874+0000","last_unstale":"2024-03-26T21:53:55.375908+0000","last_undegraded":"2024-03-26T21:53:55.375908+0000","last_fullsized":"2024-03-26T21:53:55.375908+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.653366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.653366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.129353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":1124,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393357+0000","last_change":"2024-03-26T21:53:41.819511+0000","last_active":"2024-03-26T21:53:55.393357+0000","last_peered":"2024-03-26T21:53:55.393357+0000","last_clean":"2024-03-26T21:53:55.393357+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:53:55.393357+0000","last_undegraded":"2024-03-26T21:53:55.393357+0000","last_fullsized":"2024-03-26T21:53:55.393357+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:41.819382+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.819382+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.581578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642887+0000","last_change":"2024-03-26T21:53:05.346212+0000","last_active":"2024-03-26T21:53:58.642887+0000","last_peered":"2024-03-26T21:53:58.642887+0000","last_clean":"2024-03-26T21:53:58.642887+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:53:58.642887+0000","last_undegraded":"2024-03-26T21:53:58.642887+0000","last_fullsized":"2024-03-26T21:53:58.642887+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:05.346112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:05.346112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:10.311996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"505'15","reported_seq":516,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643987+0000","last_change":"2024-03-26T21:52:44.140183+0000","last_active":"2024-03-26T21:53:58.643987+0000","last_peered":"2024-03-26T21:53:58.643987+0000","last_clean":"2024-03-26T21:53:58.643987+0000","last_became_active":"2024-03-26T21:52:08.303917+0000","last_became_peered":"2024-03-26T21:52:08.303917+0000","last_unstale":"2024-03-26T21:53:58.643987+0000","last_undegraded":"2024-03-26T21:53:58.643987+0000","last_fullsized":"2024-03-26T21:53:58.643987+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'15","last_scrub_stamp":"2024-03-26T21:52:44.140065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:44.140065+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:58.988615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":749,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423308+0000","last_change":"2024-03-26T21:53:32.760301+0000","last_active":"2024-03-26T21:53:59.423308+0000","last_peered":"2024-03-26T21:53:59.423308+0000","last_clean":"2024-03-26T21:53:59.423308+0000","last_became_active":"2024-03-26T21:52:08.295768+0000","last_became_peered":"2024-03-26T21:52:08.295768+0000","last_unstale":"2024-03-26T21:53:59.423308+0000","last_undegraded":"2024-03-26T21:53:59.423308+0000","last_fullsized":"2024-03-26T21:53:59.423308+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:32.760194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:02.948417+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.760194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:59.759274+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":416,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643806+0000","last_change":"2024-03-26T21:53:22.301307+0000","last_active":"2024-03-26T21:53:58.643806+0000","last_peered":"2024-03-26T21:53:58.643806+0000","last_clean":"2024-03-26T21:53:58.643806+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:53:58.643806+0000","last_undegraded":"2024-03-26T21:53:58.643806+0000","last_fullsized":"2024-03-26T21:53:58.643806+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-26T21:53:22.301180+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:53:22.301180+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.738037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":994,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644974+0000","last_change":"2024-03-26T21:53:03.498814+0000","last_active":"2024-03-26T21:53:58.644974+0000","last_peered":"2024-03-26T21:53:58.644974+0000","last_clean":"2024-03-26T21:53:58.644974+0000","last_became_active":"2024-03-26T21:52:08.305000+0000","last_became_peered":"2024-03-26T21:52:08.305000+0000","last_unstale":"2024-03-26T21:53:58.644974+0000","last_undegraded":"2024-03-26T21:53:58.644974+0000","last_fullsized":"2024-03-26T21:53:58.644974+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.498714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.498714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:04.371324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1349,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393165+0000","last_change":"2024-03-26T21:52:26.747492+0000","last_active":"2024-03-26T21:53:55.393165+0000","last_peered":"2024-03-26T21:53:55.393165+0000","last_clean":"2024-03-26T21:53:55.393165+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:53:55.393165+0000","last_undegraded":"2024-03-26T21:53:55.393165+0000","last_fullsized":"2024-03-26T21:53:55.393165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:52:26.747371+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:52:26.747371+0000","last_clean_scrub_stamp":"2024-03-26T21:52:26.747371+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":1047,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643461+0000","last_change":"2024-03-26T21:53:12.461418+0000","last_active":"2024-03-26T21:53:58.643461+0000","last_peered":"2024-03-26T21:53:58.643461+0000","last_clean":"2024-03-26T21:53:58.643461+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:53:58.643461+0000","last_undegraded":"2024-03-26T21:53:58.643461+0000","last_fullsized":"2024-03-26T21:53:58.643461+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:53:12.461268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.461268+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.000302+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":779,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422546+0000","last_change":"2024-03-26T21:53:27.667095+0000","last_active":"2024-03-26T21:53:59.422546+0000","last_peered":"2024-03-26T21:53:59.422546+0000","last_clean":"2024-03-26T21:53:59.422546+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:53:59.422546+0000","last_undegraded":"2024-03-26T21:53:59.422546+0000","last_fullsized":"2024-03-26T21:53:59.422546+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:27.666957+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:53:27.666957+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.271390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":739,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422609+0000","last_change":"2024-03-26T21:52:54.729667+0000","last_active":"2024-03-26T21:53:59.422609+0000","last_peered":"2024-03-26T21:53:59.422609+0000","last_clean":"2024-03-26T21:53:59.422609+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:53:59.422609+0000","last_undegraded":"2024-03-26T21:53:59.422609+0000","last_fullsized":"2024-03-26T21:53:59.422609+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:54.729584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:52:54.729584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.427352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1071,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415014+0000","last_change":"2024-03-26T21:53:48.325613+0000","last_active":"2024-03-26T21:53:58.415014+0000","last_peered":"2024-03-26T21:53:58.415014+0000","last_clean":"2024-03-26T21:53:58.415014+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:53:58.415014+0000","last_undegraded":"2024-03-26T21:53:58.415014+0000","last_fullsized":"2024-03-26T21:53:58.415014+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:48.325518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:53:48.325518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:52.642317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":389,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642707+0000","last_change":"2024-03-26T21:53:31.214121+0000","last_active":"2024-03-26T21:53:58.642707+0000","last_peered":"2024-03-26T21:53:58.642707+0000","last_clean":"2024-03-26T21:53:58.642707+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:53:58.642707+0000","last_undegraded":"2024-03-26T21:53:58.642707+0000","last_fullsized":"2024-03-26T21:53:58.642707+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:31.214018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:53:31.214018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.213833+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":1121,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415118+0000","last_change":"2024-03-26T21:53:34.337707+0000","last_active":"2024-03-26T21:53:58.415118+0000","last_peered":"2024-03-26T21:53:58.415118+0000","last_clean":"2024-03-26T21:53:58.415118+0000","last_became_active":"2024-03-26T21:52:08.312344+0000","last_became_peered":"2024-03-26T21:52:08.312344+0000","last_unstale":"2024-03-26T21:53:58.415118+0000","last_undegraded":"2024-03-26T21:53:58.415118+0000","last_fullsized":"2024-03-26T21:53:58.415118+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:34.337568+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.337568+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.277211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":987,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643547+0000","last_change":"2024-03-26T21:53:23.699793+0000","last_active":"2024-03-26T21:53:58.643547+0000","last_peered":"2024-03-26T21:53:58.643547+0000","last_clean":"2024-03-26T21:53:58.643547+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:53:58.643547+0000","last_undegraded":"2024-03-26T21:53:58.643547+0000","last_fullsized":"2024-03-26T21:53:58.643547+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:23.699681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:23.699681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:44.150085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":381,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643255+0000","last_change":"2024-03-26T21:52:23.047307+0000","last_active":"2024-03-26T21:53:58.643255+0000","last_peered":"2024-03-26T21:53:58.643255+0000","last_clean":"2024-03-26T21:53:58.643255+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:53:58.643255+0000","last_undegraded":"2024-03-26T21:53:58.643255+0000","last_fullsized":"2024-03-26T21:53:58.643255+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:23.047219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:23.047219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1027,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642359+0000","last_change":"2024-03-26T21:53:35.383088+0000","last_active":"2024-03-26T21:53:58.642359+0000","last_peered":"2024-03-26T21:53:58.642359+0000","last_clean":"2024-03-26T21:53:58.642359+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:53:58.642359+0000","last_undegraded":"2024-03-26T21:53:58.642359+0000","last_fullsized":"2024-03-26T21:53:58.642359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.382983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.382983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:53.425131+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":413,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642820+0000","last_change":"2024-03-26T21:52:50.160918+0000","last_active":"2024-03-26T21:53:58.642820+0000","last_peered":"2024-03-26T21:53:58.642820+0000","last_clean":"2024-03-26T21:53:58.642820+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:53:58.642820+0000","last_undegraded":"2024-03-26T21:53:58.642820+0000","last_fullsized":"2024-03-26T21:53:58.642820+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-26T21:52:50.160698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.160698+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1060,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392869+0000","last_change":"2024-03-26T21:53:24.786646+0000","last_active":"2024-03-26T21:53:55.392869+0000","last_peered":"2024-03-26T21:53:55.392869+0000","last_clean":"2024-03-26T21:53:55.392869+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:53:55.392869+0000","last_undegraded":"2024-03-26T21:53:55.392869+0000","last_fullsized":"2024-03-26T21:53:55.392869+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:24.786545+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.804313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":964,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644388+0000","last_change":"2024-03-26T21:53:40.537631+0000","last_active":"2024-03-26T21:53:58.644388+0000","last_peered":"2024-03-26T21:53:58.644388+0000","last_clean":"2024-03-26T21:53:58.644388+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:53:58.644388+0000","last_undegraded":"2024-03-26T21:53:58.644388+0000","last_fullsized":"2024-03-26T21:53:58.644388+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:40.537460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:40.537460+0000","last_clean_scrub_stamp":"2024-03-26T21:53:40.537460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.728488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":389,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643350+0000","last_change":"2024-03-26T21:52:41.284391+0000","last_active":"2024-03-26T21:53:58.643350+0000","last_peered":"2024-03-26T21:53:58.643350+0000","last_clean":"2024-03-26T21:53:58.643350+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:53:58.643350+0000","last_undegraded":"2024-03-26T21:53:58.643350+0000","last_fullsized":"2024-03-26T21:53:58.643350+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:41.284262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:52:41.284262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:00.239810+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":401,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642536+0000","last_change":"2024-03-26T21:52:39.203924+0000","last_active":"2024-03-26T21:53:58.642536+0000","last_peered":"2024-03-26T21:53:58.642536+0000","last_clean":"2024-03-26T21:53:58.642536+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:53:58.642536+0000","last_undegraded":"2024-03-26T21:53:58.642536+0000","last_fullsized":"2024-03-26T21:53:58.642536+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-03-26T21:52:39.203836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:52:39.203836+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1051,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643349+0000","last_change":"2024-03-26T21:53:40.265060+0000","last_active":"2024-03-26T21:53:58.643349+0000","last_peered":"2024-03-26T21:53:58.643349+0000","last_clean":"2024-03-26T21:53:58.643349+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:53:58.643349+0000","last_undegraded":"2024-03-26T21:53:58.643349+0000","last_fullsized":"2024-03-26T21:53:58.643349+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:40.264945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:40.264945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.943230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":940,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393121+0000","last_change":"2024-03-26T21:53:02.593369+0000","last_active":"2024-03-26T21:53:55.393121+0000","last_peered":"2024-03-26T21:53:55.393121+0000","last_clean":"2024-03-26T21:53:55.393121+0000","last_became_active":"2024-03-26T21:53:02.593001+0000","last_became_peered":"2024-03-26T21:53:02.593001+0000","last_unstale":"2024-03-26T21:53:55.393121+0000","last_undegraded":"2024-03-26T21:53:55.393121+0000","last_fullsized":"2024-03-26T21:53:55.393121+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:24.762018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:52:24.762018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":984,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643083+0000","last_change":"2024-03-26T21:53:20.690699+0000","last_active":"2024-03-26T21:53:58.643083+0000","last_peered":"2024-03-26T21:53:58.643083+0000","last_clean":"2024-03-26T21:53:58.643083+0000","last_became_active":"2024-03-26T21:52:08.296169+0000","last_became_peered":"2024-03-26T21:52:08.296169+0000","last_unstale":"2024-03-26T21:53:58.643083+0000","last_undegraded":"2024-03-26T21:53:58.643083+0000","last_fullsized":"2024-03-26T21:53:58.643083+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:20.690575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:20.690575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:30.897935+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":980,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393321+0000","last_change":"2024-03-26T21:53:18.354751+0000","last_active":"2024-03-26T21:53:55.393321+0000","last_peered":"2024-03-26T21:53:55.393321+0000","last_clean":"2024-03-26T21:53:55.393321+0000","last_became_active":"2024-03-26T21:53:02.594695+0000","last_became_peered":"2024-03-26T21:53:02.594695+0000","last_unstale":"2024-03-26T21:53:55.393321+0000","last_undegraded":"2024-03-26T21:53:55.393321+0000","last_fullsized":"2024-03-26T21:53:55.393321+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:18.354628+0000","last_deep_scrub":"456'32","last_deep_scrub_stamp":"2024-03-26T21:52:04.722406+0000","last_clean_scrub_stamp":"2024-03-26T21:53:18.354628+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.280649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":381,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642429+0000","last_change":"2024-03-26T21:52:37.133446+0000","last_active":"2024-03-26T21:53:58.642429+0000","last_peered":"2024-03-26T21:53:58.642429+0000","last_clean":"2024-03-26T21:53:58.642429+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:53:58.642429+0000","last_undegraded":"2024-03-26T21:53:58.642429+0000","last_fullsized":"2024-03-26T21:53:58.642429+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:37.133301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:52:37.133301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643779+0000","last_change":"2024-03-26T21:52:57.406338+0000","last_active":"2024-03-26T21:53:58.643779+0000","last_peered":"2024-03-26T21:53:58.643779+0000","last_clean":"2024-03-26T21:53:58.643779+0000","last_became_active":"2024-03-26T21:52:08.299914+0000","last_became_peered":"2024-03-26T21:52:08.299914+0000","last_unstale":"2024-03-26T21:53:58.643779+0000","last_undegraded":"2024-03-26T21:53:58.643779+0000","last_fullsized":"2024-03-26T21:53:58.643779+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:57.406234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:52:57.406234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:14.651364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1064,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415232+0000","last_change":"2024-03-26T21:53:55.595005+0000","last_active":"2024-03-26T21:53:58.415232+0000","last_peered":"2024-03-26T21:53:58.415232+0000","last_clean":"2024-03-26T21:53:58.415232+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:53:58.415232+0000","last_undegraded":"2024-03-26T21:53:58.415232+0000","last_fullsized":"2024-03-26T21:53:58.415232+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:55.594874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:36.551754+0000","last_clean_scrub_stamp":"2024-03-26T21:53:55.594874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:14.351591+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643663+0000","last_change":"2024-03-26T21:53:44.295771+0000","last_active":"2024-03-26T21:53:58.643663+0000","last_peered":"2024-03-26T21:53:58.643663+0000","last_clean":"2024-03-26T21:53:58.643663+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:53:58.643663+0000","last_undegraded":"2024-03-26T21:53:58.643663+0000","last_fullsized":"2024-03-26T21:53:58.643663+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-26T21:53:44.295661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:44.295661+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:54.251014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":1098,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.414750+0000","last_change":"2024-03-26T21:53:30.330738+0000","last_active":"2024-03-26T21:53:58.414750+0000","last_peered":"2024-03-26T21:53:58.414750+0000","last_clean":"2024-03-26T21:53:58.414750+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:53:58.414750+0000","last_undegraded":"2024-03-26T21:53:58.414750+0000","last_fullsized":"2024-03-26T21:53:58.414750+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:53:30.330651+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.330651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.444222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"531'16","reported_seq":1227,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393697+0000","last_change":"2024-03-26T21:53:02.593365+0000","last_active":"2024-03-26T21:53:55.393697+0000","last_peered":"2024-03-26T21:53:55.393697+0000","last_clean":"2024-03-26T21:53:55.393697+0000","last_became_active":"2024-03-26T21:53:02.592915+0000","last_became_peered":"2024-03-26T21:53:02.592915+0000","last_unstale":"2024-03-26T21:53:55.393697+0000","last_undegraded":"2024-03-26T21:53:55.393697+0000","last_fullsized":"2024-03-26T21:53:55.393697+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'15","last_scrub_stamp":"2024-03-26T21:52:50.727708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.727708+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.583946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":988,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644337+0000","last_change":"2024-03-26T21:53:21.676383+0000","last_active":"2024-03-26T21:53:58.644337+0000","last_peered":"2024-03-26T21:53:58.644337+0000","last_clean":"2024-03-26T21:53:58.644337+0000","last_became_active":"2024-03-26T21:52:08.297191+0000","last_became_peered":"2024-03-26T21:52:08.297191+0000","last_unstale":"2024-03-26T21:53:58.644337+0000","last_undegraded":"2024-03-26T21:53:58.644337+0000","last_fullsized":"2024-03-26T21:53:58.644337+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:53:21.676270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:21.676270+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.217592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":415,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643321+0000","last_change":"2024-03-26T21:52:51.140833+0000","last_active":"2024-03-26T21:53:58.643321+0000","last_peered":"2024-03-26T21:53:58.643321+0000","last_clean":"2024-03-26T21:53:58.643321+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:53:58.643321+0000","last_undegraded":"2024-03-26T21:53:58.643321+0000","last_fullsized":"2024-03-26T21:53:58.643321+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-26T21:52:51.140738+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-26T21:52:51.140738+0000","last_clean_scrub_stamp":"2024-03-26T21:52:51.140738+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:08.973250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":749,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422279+0000","last_change":"2024-03-26T21:53:55.711499+0000","last_active":"2024-03-26T21:53:59.422279+0000","last_peered":"2024-03-26T21:53:59.422279+0000","last_clean":"2024-03-26T21:53:59.422279+0000","last_became_active":"2024-03-26T21:52:08.303451+0000","last_became_peered":"2024-03-26T21:52:08.303451+0000","last_unstale":"2024-03-26T21:53:59.422279+0000","last_undegraded":"2024-03-26T21:53:59.422279+0000","last_fullsized":"2024-03-26T21:53:59.422279+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:55.711401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:55.711401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.060875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":773,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422943+0000","last_change":"2024-03-26T21:53:39.562066+0000","last_active":"2024-03-26T21:53:59.422943+0000","last_peered":"2024-03-26T21:53:59.422943+0000","last_clean":"2024-03-26T21:53:59.422943+0000","last_became_active":"2024-03-26T21:52:08.303646+0000","last_became_peered":"2024-03-26T21:52:08.303646+0000","last_unstale":"2024-03-26T21:53:59.422943+0000","last_undegraded":"2024-03-26T21:53:59.422943+0000","last_fullsized":"2024-03-26T21:53:59.422943+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:39.561927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.561927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.143555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":382,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.644105+0000","last_change":"2024-03-26T21:52:19.244877+0000","last_active":"2024-03-26T21:53:58.644105+0000","last_peered":"2024-03-26T21:53:58.644105+0000","last_clean":"2024-03-26T21:53:58.644105+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:53:58.644105+0000","last_undegraded":"2024-03-26T21:53:58.644105+0000","last_fullsized":"2024-03-26T21:53:58.644105+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:19.244767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:19.244767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"531'46","reported_seq":110,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.375781+0000","last_change":"2024-03-26T21:53:06.705275+0000","last_active":"2024-03-26T21:53:55.375781+0000","last_peered":"2024-03-26T21:53:55.375781+0000","last_clean":"2024-03-26T21:53:55.375781+0000","last_became_active":"2024-03-26T21:52:08.306770+0000","last_became_peered":"2024-03-26T21:52:08.306770+0000","last_unstale":"2024-03-26T21:53:55.375781+0000","last_undegraded":"2024-03-26T21:53:55.375781+0000","last_fullsized":"2024-03-26T21:53:55.375781+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'46","last_scrub_stamp":"2024-03-26T21:53:06.705083+0000","last_deep_scrub":"531'46","last_deep_scrub_stamp":"2024-03-26T21:53:06.705083+0000","last_clean_scrub_stamp":"2024-03-26T21:53:06.705083+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.738937+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643276+0000","last_change":"2024-03-26T21:53:03.375639+0000","last_active":"2024-03-26T21:53:58.643276+0000","last_peered":"2024-03-26T21:53:58.643276+0000","last_clean":"2024-03-26T21:53:58.643276+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:53:58.643276+0000","last_undegraded":"2024-03-26T21:53:58.643276+0000","last_fullsized":"2024-03-26T21:53:58.643276+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.375556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.375556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:15.145507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1060,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.415480+0000","last_change":"2024-03-26T21:53:41.251121+0000","last_active":"2024-03-26T21:53:58.415480+0000","last_peered":"2024-03-26T21:53:58.415480+0000","last_clean":"2024-03-26T21:53:58.415480+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:53:58.415480+0000","last_undegraded":"2024-03-26T21:53:58.415480+0000","last_fullsized":"2024-03-26T21:53:58.415480+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:41.251038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.251038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:09.822942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":731,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423162+0000","last_change":"2024-03-26T21:52:49.747504+0000","last_active":"2024-03-26T21:53:59.423162+0000","last_peered":"2024-03-26T21:53:59.423162+0000","last_clean":"2024-03-26T21:53:59.423162+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:53:59.423162+0000","last_undegraded":"2024-03-26T21:53:59.423162+0000","last_fullsized":"2024-03-26T21:53:59.423162+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:49.747390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:49.747390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:11.171018+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":773,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422618+0000","last_change":"2024-03-26T21:53:43.598272+0000","last_active":"2024-03-26T21:53:59.422618+0000","last_peered":"2024-03-26T21:53:59.422618+0000","last_clean":"2024-03-26T21:53:59.422618+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:53:59.422618+0000","last_undegraded":"2024-03-26T21:53:59.422618+0000","last_fullsized":"2024-03-26T21:53:59.422618+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-26T21:53:43.598165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:43.598165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.410698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1064,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393325+0000","last_change":"2024-03-26T21:52:14.710152+0000","last_active":"2024-03-26T21:53:55.393325+0000","last_peered":"2024-03-26T21:53:55.393325+0000","last_clean":"2024-03-26T21:53:55.393325+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:53:55.393325+0000","last_undegraded":"2024-03-26T21:53:55.393325+0000","last_fullsized":"2024-03-26T21:53:55.393325+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:14.710054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:52:14.710054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642792+0000","last_change":"2024-03-26T21:53:08.390211+0000","last_active":"2024-03-26T21:53:58.642792+0000","last_peered":"2024-03-26T21:53:58.642792+0000","last_clean":"2024-03-26T21:53:58.642792+0000","last_became_active":"2024-03-26T21:51:57.556255+0000","last_became_peered":"2024-03-26T21:51:57.556255+0000","last_unstale":"2024-03-26T21:53:58.642792+0000","last_undegraded":"2024-03-26T21:53:58.642792+0000","last_fullsized":"2024-03-26T21:53:58.642792+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:08.390084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:08.390084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:34.423654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1051,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392606+0000","last_change":"2024-03-26T21:53:19.868514+0000","last_active":"2024-03-26T21:53:55.392606+0000","last_peered":"2024-03-26T21:53:55.392606+0000","last_clean":"2024-03-26T21:53:55.392606+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:53:55.392606+0000","last_undegraded":"2024-03-26T21:53:55.392606+0000","last_fullsized":"2024-03-26T21:53:55.392606+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:19.868402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.868402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.602040+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":797,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422585+0000","last_change":"2024-03-26T21:53:00.655476+0000","last_active":"2024-03-26T21:53:59.422585+0000","last_peered":"2024-03-26T21:53:59.422585+0000","last_clean":"2024-03-26T21:53:59.422585+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:53:59.422585+0000","last_undegraded":"2024-03-26T21:53:59.422585+0000","last_fullsized":"2024-03-26T21:53:59.422585+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:00.655364+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:53:00.655364+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.067394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1072,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.393074+0000","last_change":"2024-03-26T21:53:35.795881+0000","last_active":"2024-03-26T21:53:55.393074+0000","last_peered":"2024-03-26T21:53:55.393074+0000","last_clean":"2024-03-26T21:53:55.393074+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:53:55.393074+0000","last_undegraded":"2024-03-26T21:53:55.393074+0000","last_fullsized":"2024-03-26T21:53:55.393074+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.795792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.795792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.161500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1044,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392756+0000","last_change":"2024-03-26T21:52:10.695486+0000","last_active":"2024-03-26T21:53:55.392756+0000","last_peered":"2024-03-26T21:53:55.392756+0000","last_clean":"2024-03-26T21:53:55.392756+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:53:55.392756+0000","last_undegraded":"2024-03-26T21:53:55.392756+0000","last_fullsized":"2024-03-26T21:53:55.392756+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:10.695287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_clean_scrub_stamp":"2024-03-26T21:52:10.695287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642602+0000","last_change":"2024-03-26T21:53:24.430260+0000","last_active":"2024-03-26T21:53:58.642602+0000","last_peered":"2024-03-26T21:53:58.642602+0000","last_clean":"2024-03-26T21:53:58.642602+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:53:58.642602+0000","last_undegraded":"2024-03-26T21:53:58.642602+0000","last_fullsized":"2024-03-26T21:53:58.642602+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.430140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.430140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:32.565270+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":101,"reported_epoch":539,"state":"active+clean","last_fresh":"2024-03-26T21:53:55.392424+0000","last_change":"2024-03-26T21:53:42.785539+0000","last_active":"2024-03-26T21:53:55.392424+0000","last_peered":"2024-03-26T21:53:55.392424+0000","last_clean":"2024-03-26T21:53:55.392424+0000","last_became_active":"2024-03-26T21:52:17.976213+0000","last_became_peered":"2024-03-26T21:52:17.976213+0000","last_unstale":"2024-03-26T21:53:55.392424+0000","last_undegraded":"2024-03-26T21:53:55.392424+0000","last_fullsized":"2024-03-26T21:53:55.392424+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:53:42.785401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:42.785401+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:53.180851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":402,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643907+0000","last_change":"2024-03-26T21:53:42.153289+0000","last_active":"2024-03-26T21:53:58.643907+0000","last_peered":"2024-03-26T21:53:58.643907+0000","last_clean":"2024-03-26T21:53:58.643907+0000","last_became_active":"2024-03-26T21:52:08.303017+0000","last_became_peered":"2024-03-26T21:52:08.303017+0000","last_unstale":"2024-03-26T21:53:58.643907+0000","last_undegraded":"2024-03-26T21:53:58.643907+0000","last_fullsized":"2024-03-26T21:53:58.643907+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:42.153207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:42.153207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.008511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":738,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.421873+0000","last_change":"2024-03-26T21:53:22.680583+0000","last_active":"2024-03-26T21:53:59.421873+0000","last_peered":"2024-03-26T21:53:59.421873+0000","last_clean":"2024-03-26T21:53:59.421873+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:53:59.421873+0000","last_undegraded":"2024-03-26T21:53:59.421873+0000","last_fullsized":"2024-03-26T21:53:59.421873+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:22.680471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:22.680471+0000","last_clean_scrub_stamp":"2024-03-26T21:53:22.680471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.861277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":389,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643144+0000","last_change":"2024-03-26T21:53:30.378126+0000","last_active":"2024-03-26T21:53:58.643144+0000","last_peered":"2024-03-26T21:53:58.643144+0000","last_clean":"2024-03-26T21:53:58.643144+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:53:58.643144+0000","last_undegraded":"2024-03-26T21:53:58.643144+0000","last_fullsized":"2024-03-26T21:53:58.643144+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:30.378020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.378020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:34.529284+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":782,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422008+0000","last_change":"2024-03-26T21:53:33.622669+0000","last_active":"2024-03-26T21:53:59.422008+0000","last_peered":"2024-03-26T21:53:59.422008+0000","last_clean":"2024-03-26T21:53:59.422008+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:53:59.422008+0000","last_undegraded":"2024-03-26T21:53:59.422008+0000","last_fullsized":"2024-03-26T21:53:59.422008+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-26T21:53:33.622527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:33.622527+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:40.365986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643136+0000","last_change":"2024-03-26T21:53:28.377735+0000","last_active":"2024-03-26T21:53:58.643136+0000","last_peered":"2024-03-26T21:53:58.643136+0000","last_clean":"2024-03-26T21:53:58.643136+0000","last_became_active":"2024-03-26T21:52:08.299872+0000","last_became_peered":"2024-03-26T21:52:08.299872+0000","last_unstale":"2024-03-26T21:53:58.643136+0000","last_undegraded":"2024-03-26T21:53:58.643136+0000","last_fullsized":"2024-03-26T21:53:58.643136+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:28.377651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:27.414633+0000","last_clean_scrub_stamp":"2024-03-26T21:53:28.377651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:54.328768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1048,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.414495+0000","last_change":"2024-03-26T21:53:23.273559+0000","last_active":"2024-03-26T21:53:58.414495+0000","last_peered":"2024-03-26T21:53:58.414495+0000","last_clean":"2024-03-26T21:53:58.414495+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:53:58.414495+0000","last_undegraded":"2024-03-26T21:53:58.414495+0000","last_fullsized":"2024-03-26T21:53:58.414495+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:23.273480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:23.273480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.468181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":737,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422412+0000","last_change":"2024-03-26T21:53:01.728655+0000","last_active":"2024-03-26T21:53:59.422412+0000","last_peered":"2024-03-26T21:53:59.422412+0000","last_clean":"2024-03-26T21:53:59.422412+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:53:59.422412+0000","last_undegraded":"2024-03-26T21:53:59.422412+0000","last_fullsized":"2024-03-26T21:53:59.422412+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.728552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:01.728552+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.728552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:11.299636+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1402,"num_read_kb":1397,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":8038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1098,"ondisk_log_size":1098,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973849,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596396,"kb_used_data":1316,"kb_used_omap":6,"kb_used_meta":594937,"kb_avail":93148756,"statfs":{"total":95995035648,"available":95384326144,"internally_reserved":0,"allocated":1347584,"data_stored":953815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":609215784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365116,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93012852,"statfs":{"total":95995035648,"available":95245160448,"internally_reserved":0,"allocated":1314816,"data_stored":951403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575190,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974576,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92770576,"statfs":{"total":95995035648,"available":94997069824,"internally_reserved":0,"allocated":757760,"data_stored":362458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738715,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939064,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806088,"statfs":{"total":95995035648,"available":95033434112,"internally_reserved":0,"allocated":1359872,"data_stored":953451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411664,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735756,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009396,"statfs":{"total":95995035648,"available":95241621504,"internally_reserved":0,"allocated":720896,"data_stored":361260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150026,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988464,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756688,"statfs":{"total":95995035648,"available":94982848512,"internally_reserved":0,"allocated":757760,"data_stored":361829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650623,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817288,"kb_used_data":764,"kb_used_omap":4,"kb_used_meta":816507,"kb_avail":92927864,"statfs":{"total":95995035648,"available":95158132736,"internally_reserved":0,"allocated":782336,"data_stored":376433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":836103843},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659455,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735904,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009248,"statfs":{"total":95995035648,"available":95241469952,"internally_reserved":0,"allocated":745472,"data_stored":361829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:54:02.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T21:54:03.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-26T21:54:03.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:03.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: 5.5 scrub starts 2024-03-26T21:54:03.491 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: 5.5 scrub ok 2024-03-26T21:54:03.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: 3.18 scrub starts 2024-03-26T21:54:03.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: 3.18 scrub ok 2024-03-26T21:54:03.492 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:03 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:03.499 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:54:03.764 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: 5.5 scrub starts 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: 5.5 scrub ok 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: 3.18 scrub starts 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: 3.18 scrub ok 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: from='client.16266 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:03.765 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: 5.5 scrub starts 2024-03-26T21:54:03.766 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: 5.5 scrub ok 2024-03-26T21:54:03.766 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: 3.18 scrub starts 2024-03-26T21:54:03.766 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: 3.18 scrub ok 2024-03-26T21:54:03.766 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:03 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T21:54:04.569 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:04 smithi055 ceph-mon[40587]: 3.d deep-scrub starts 2024-03-26T21:54:04.578 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:04 smithi055 ceph-mon[40587]: 3.d deep-scrub ok 2024-03-26T21:54:04.579 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:04 smithi055 ceph-mon[29406]: 3.d deep-scrub starts 2024-03-26T21:54:04.579 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:04 smithi055 ceph-mon[29406]: 3.d deep-scrub ok 2024-03-26T21:54:04.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:04 smithi067 ceph-mon[33346]: 3.d deep-scrub starts 2024-03-26T21:54:04.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:04 smithi067 ceph-mon[33346]: 3.d deep-scrub ok 2024-03-26T21:54:05.554 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-26T21:54:05.556 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2024-03-26T21:54:05.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:05 smithi067 ceph-mon[33346]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:05.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:05 smithi067 ceph-mon[33346]: 2.2 deep-scrub starts 2024-03-26T21:54:05.741 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:05 smithi067 ceph-mon[33346]: 2.2 deep-scrub ok 2024-03-26T21:54:05.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[29406]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:05.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[29406]: 2.2 deep-scrub starts 2024-03-26T21:54:05.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[29406]: 2.2 deep-scrub ok 2024-03-26T21:54:05.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[40587]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:05.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[40587]: 2.2 deep-scrub starts 2024-03-26T21:54:05.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:05 smithi055 ceph-mon[40587]: 2.2 deep-scrub ok 2024-03-26T21:54:06.674 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1206,"stamp":"2024-03-26T21:54:03.838175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3151,"num_read_kb":2999,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1780184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1349,"ondisk_log_size":1349,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6519524,"kb_used_data":7748,"kb_used_omap":20,"kb_used_meta":6511659,"kb_avail":743441692,"statfs":{"total":767960285184,"available":761284292608,"internally_reserved":0,"allocated":7933952,"data_stored":4763154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20884,"internal_metadata":6667939436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001790"},"pg_stats":[{"pgid":"4.c","version":"456'35","reported_seq":852,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422746+0000","last_change":"2024-03-26T21:53:37.605343+0000","last_active":"2024-03-26T21:53:59.422746+0000","last_peered":"2024-03-26T21:53:59.422746+0000","last_clean":"2024-03-26T21:53:59.422746+0000","last_became_active":"2024-03-26T21:52:17.974507+0000","last_became_peered":"2024-03-26T21:52:17.974507+0000","last_unstale":"2024-03-26T21:53:59.422746+0000","last_undegraded":"2024-03-26T21:53:59.422746+0000","last_fullsized":"2024-03-26T21:53:59.422746+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-26T21:53:37.605172+0000","last_deep_scrub":"234'25","last_deep_scrub_stamp":"2024-03-26T21:43:08.600649+0000","last_clean_scrub_stamp":"2024-03-26T21:53:37.605172+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.319108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":392,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665316+0000","last_change":"2024-03-26T21:53:12.156545+0000","last_active":"2024-03-26T21:53:59.665316+0000","last_peered":"2024-03-26T21:53:59.665316+0000","last_clean":"2024-03-26T21:53:59.665316+0000","last_became_active":"2024-03-26T21:44:49.408765+0000","last_became_peered":"2024-03-26T21:44:49.408765+0000","last_unstale":"2024-03-26T21:53:59.665316+0000","last_undegraded":"2024-03-26T21:53:59.665316+0000","last_fullsized":"2024-03-26T21:53:59.665316+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:12.156464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.156464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.708287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"505'13","reported_seq":126,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424708+0000","last_change":"2024-03-26T21:53:18.684219+0000","last_active":"2024-03-26T21:53:59.424708+0000","last_peered":"2024-03-26T21:53:59.424708+0000","last_clean":"2024-03-26T21:53:59.424708+0000","last_became_active":"2024-03-26T21:52:08.293748+0000","last_became_peered":"2024-03-26T21:52:08.293748+0000","last_unstale":"2024-03-26T21:53:59.424708+0000","last_undegraded":"2024-03-26T21:53:59.424708+0000","last_fullsized":"2024-03-26T21:53:59.424708+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'13","last_scrub_stamp":"2024-03-26T21:53:18.684110+0000","last_deep_scrub":"235'9","last_deep_scrub_stamp":"2024-03-26T21:38:57.799649+0000","last_clean_scrub_stamp":"2024-03-26T21:53:18.684110+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.560249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":751,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423040+0000","last_change":"2024-03-26T21:53:31.694080+0000","last_active":"2024-03-26T21:53:59.423040+0000","last_peered":"2024-03-26T21:53:59.423040+0000","last_clean":"2024-03-26T21:53:59.423040+0000","last_became_active":"2024-03-26T21:52:08.306888+0000","last_became_peered":"2024-03-26T21:52:08.306888+0000","last_unstale":"2024-03-26T21:53:59.423040+0000","last_undegraded":"2024-03-26T21:53:59.423040+0000","last_fullsized":"2024-03-26T21:53:59.423040+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:31.693947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:41:56.817883+0000","last_clean_scrub_stamp":"2024-03-26T21:53:31.693947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:43.995259+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":854,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423081+0000","last_change":"2024-03-26T21:53:17.682610+0000","last_active":"2024-03-26T21:53:59.423081+0000","last_peered":"2024-03-26T21:53:59.423081+0000","last_clean":"2024-03-26T21:53:59.423081+0000","last_became_active":"2024-03-26T21:52:08.306232+0000","last_became_peered":"2024-03-26T21:52:08.306232+0000","last_unstale":"2024-03-26T21:53:59.423081+0000","last_undegraded":"2024-03-26T21:53:59.423081+0000","last_fullsized":"2024-03-26T21:53:59.423081+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-26T21:53:17.682481+0000","last_deep_scrub":"282'39","last_deep_scrub_stamp":"2024-03-26T21:41:50.858942+0000","last_clean_scrub_stamp":"2024-03-26T21:53:17.682481+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.828789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":952,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822403+0000","last_change":"2024-03-26T21:53:04.375087+0000","last_active":"2024-03-26T21:53:59.822403+0000","last_peered":"2024-03-26T21:53:59.822403+0000","last_clean":"2024-03-26T21:53:59.822403+0000","last_became_active":"2024-03-26T21:53:02.589212+0000","last_became_peered":"2024-03-26T21:53:02.589212+0000","last_unstale":"2024-03-26T21:53:59.822403+0000","last_undegraded":"2024-03-26T21:53:59.822403+0000","last_fullsized":"2024-03-26T21:53:59.822403+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:04.374971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:33.006101+0000","last_clean_scrub_stamp":"2024-03-26T21:53:04.374971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:07.489077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1071,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425903+0000","last_change":"2024-03-26T21:53:14.411181+0000","last_active":"2024-03-26T21:53:59.425903+0000","last_peered":"2024-03-26T21:53:59.425903+0000","last_clean":"2024-03-26T21:53:59.425903+0000","last_became_active":"2024-03-26T21:36:51.682178+0000","last_became_peered":"2024-03-26T21:36:51.682178+0000","last_unstale":"2024-03-26T21:53:59.425903+0000","last_undegraded":"2024-03-26T21:53:59.425903+0000","last_fullsized":"2024-03-26T21:53:59.425903+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:14.411072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:14.411072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.859704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":99,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424859+0000","last_change":"2024-03-26T21:53:20.858847+0000","last_active":"2024-03-26T21:53:59.424859+0000","last_peered":"2024-03-26T21:53:59.424859+0000","last_clean":"2024-03-26T21:53:59.424859+0000","last_became_active":"2024-03-26T21:52:08.304158+0000","last_became_peered":"2024-03-26T21:52:08.304158+0000","last_unstale":"2024-03-26T21:53:59.424859+0000","last_undegraded":"2024-03-26T21:53:59.424859+0000","last_fullsized":"2024-03-26T21:53:59.424859+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:20.858690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:09.649543+0000","last_clean_scrub_stamp":"2024-03-26T21:53:20.858690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:43.223047+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":779,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423056+0000","last_change":"2024-03-26T21:53:36.638936+0000","last_active":"2024-03-26T21:53:59.423056+0000","last_peered":"2024-03-26T21:53:59.423056+0000","last_clean":"2024-03-26T21:53:59.423056+0000","last_became_active":"2024-03-26T21:36:51.703988+0000","last_became_peered":"2024-03-26T21:36:51.703988+0000","last_unstale":"2024-03-26T21:53:59.423056+0000","last_undegraded":"2024-03-26T21:53:59.423056+0000","last_fullsized":"2024-03-26T21:53:59.423056+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:36.638821+0000","last_deep_scrub":"426'24","last_deep_scrub_stamp":"2024-03-26T21:47:27.572552+0000","last_clean_scrub_stamp":"2024-03-26T21:53:36.638821+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.913029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":751,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422174+0000","last_change":"2024-03-26T21:52:59.646429+0000","last_active":"2024-03-26T21:53:59.422174+0000","last_peered":"2024-03-26T21:53:59.422174+0000","last_clean":"2024-03-26T21:53:59.422174+0000","last_became_active":"2024-03-26T21:52:08.303111+0000","last_became_peered":"2024-03-26T21:52:08.303111+0000","last_unstale":"2024-03-26T21:53:59.422174+0000","last_undegraded":"2024-03-26T21:53:59.422174+0000","last_fullsized":"2024-03-26T21:53:59.422174+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:59.646315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:41.750677+0000","last_clean_scrub_stamp":"2024-03-26T21:52:59.646315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.828998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1061,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769847+0000","last_change":"2024-03-26T21:52:36.728256+0000","last_active":"2024-03-26T21:53:59.769847+0000","last_peered":"2024-03-26T21:53:59.769847+0000","last_clean":"2024-03-26T21:53:59.769847+0000","last_became_active":"2024-03-26T21:36:51.685226+0000","last_became_peered":"2024-03-26T21:36:51.685226+0000","last_unstale":"2024-03-26T21:53:59.769847+0000","last_undegraded":"2024-03-26T21:53:59.769847+0000","last_fullsized":"2024-03-26T21:53:59.769847+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:36.728175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:36.728175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":384,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665187+0000","last_change":"2024-03-26T21:52:29.072033+0000","last_active":"2024-03-26T21:53:59.665187+0000","last_peered":"2024-03-26T21:53:59.665187+0000","last_clean":"2024-03-26T21:53:59.665187+0000","last_became_active":"2024-03-26T21:44:49.390741+0000","last_became_peered":"2024-03-26T21:44:49.390741+0000","last_unstale":"2024-03-26T21:53:59.665187+0000","last_undegraded":"2024-03-26T21:53:59.665187+0000","last_fullsized":"2024-03-26T21:53:59.665187+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:29.071901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:34.924056+0000","last_clean_scrub_stamp":"2024-03-26T21:52:29.071901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1208,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425597+0000","last_change":"2024-03-26T21:53:29.289553+0000","last_active":"2024-03-26T21:53:59.425597+0000","last_peered":"2024-03-26T21:53:59.425597+0000","last_clean":"2024-03-26T21:53:59.425597+0000","last_became_active":"2024-03-26T21:44:49.437937+0000","last_became_peered":"2024-03-26T21:44:49.437937+0000","last_unstale":"2024-03-26T21:53:59.425597+0000","last_undegraded":"2024-03-26T21:53:59.425597+0000","last_fullsized":"2024-03-26T21:53:59.425597+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:53:29.289458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:29.289458+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.817050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":392,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664674+0000","last_change":"2024-03-26T21:53:59.013454+0000","last_active":"2024-03-26T21:53:59.664674+0000","last_peered":"2024-03-26T21:53:59.664674+0000","last_clean":"2024-03-26T21:53:59.664674+0000","last_became_active":"2024-03-26T21:44:49.426609+0000","last_became_peered":"2024-03-26T21:44:49.426609+0000","last_unstale":"2024-03-26T21:53:59.664674+0000","last_undegraded":"2024-03-26T21:53:59.664674+0000","last_fullsized":"2024-03-26T21:53:59.664674+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:59.013340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:59.013340+0000","last_clean_scrub_stamp":"2024-03-26T21:53:59.013340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:27.541803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"512'13","reported_seq":958,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423185+0000","last_change":"2024-03-26T21:53:38.608081+0000","last_active":"2024-03-26T21:53:59.423185+0000","last_peered":"2024-03-26T21:53:59.423185+0000","last_clean":"2024-03-26T21:53:59.423185+0000","last_became_active":"2024-03-26T21:36:51.714572+0000","last_became_peered":"2024-03-26T21:36:51.714572+0000","last_unstale":"2024-03-26T21:53:59.423185+0000","last_undegraded":"2024-03-26T21:53:59.423185+0000","last_fullsized":"2024-03-26T21:53:59.423185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'13","last_scrub_stamp":"2024-03-26T21:53:38.607961+0000","last_deep_scrub":"512'13","last_deep_scrub_stamp":"2024-03-26T21:52:25.766920+0000","last_clean_scrub_stamp":"2024-03-26T21:53:38.607961+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.096308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1044,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769203+0000","last_change":"2024-03-26T21:53:51.789465+0000","last_active":"2024-03-26T21:53:59.769203+0000","last_peered":"2024-03-26T21:53:59.769203+0000","last_clean":"2024-03-26T21:53:59.769203+0000","last_became_active":"2024-03-26T21:28:40.891242+0000","last_became_peered":"2024-03-26T21:28:40.891242+0000","last_unstale":"2024-03-26T21:53:59.769203+0000","last_undegraded":"2024-03-26T21:53:59.769203+0000","last_fullsized":"2024-03-26T21:53:59.769203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:51.789350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:24.454506+0000","last_clean_scrub_stamp":"2024-03-26T21:53:51.789350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.358896+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":758,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422396+0000","last_change":"2024-03-26T21:53:34.669331+0000","last_active":"2024-03-26T21:53:59.422396+0000","last_peered":"2024-03-26T21:53:59.422396+0000","last_clean":"2024-03-26T21:53:59.422396+0000","last_became_active":"2024-03-26T21:44:49.411186+0000","last_became_peered":"2024-03-26T21:44:49.411186+0000","last_unstale":"2024-03-26T21:53:59.422396+0000","last_undegraded":"2024-03-26T21:53:59.422396+0000","last_fullsized":"2024-03-26T21:53:59.422396+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.669230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:10.787528+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.669230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:35.981381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":428,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664531+0000","last_change":"2024-03-26T21:53:39.184126+0000","last_active":"2024-03-26T21:53:59.664531+0000","last_peered":"2024-03-26T21:53:59.664531+0000","last_clean":"2024-03-26T21:53:59.664531+0000","last_became_active":"2024-03-26T21:44:49.406937+0000","last_became_peered":"2024-03-26T21:44:49.406937+0000","last_unstale":"2024-03-26T21:53:59.664531+0000","last_undegraded":"2024-03-26T21:53:59.664531+0000","last_fullsized":"2024-03-26T21:53:59.664531+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'50","last_scrub_stamp":"2024-03-26T21:53:39.183998+0000","last_deep_scrub":"317'38","last_deep_scrub_stamp":"2024-03-26T21:42:54.348885+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.183998+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:51.122789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":392,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665744+0000","last_change":"2024-03-26T21:53:16.195641+0000","last_active":"2024-03-26T21:53:59.665744+0000","last_peered":"2024-03-26T21:53:59.665744+0000","last_clean":"2024-03-26T21:53:59.665744+0000","last_became_active":"2024-03-26T21:44:49.417678+0000","last_became_peered":"2024-03-26T21:44:49.417678+0000","last_unstale":"2024-03-26T21:53:59.665744+0000","last_undegraded":"2024-03-26T21:53:59.665744+0000","last_fullsized":"2024-03-26T21:53:59.665744+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:16.195538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:16.195538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:26.536932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":771,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.666521+0000","last_change":"2024-03-26T21:53:59.666546+0000","last_active":"2024-03-26T21:53:59.666521+0000","last_peered":"2024-03-26T21:53:59.666521+0000","last_clean":"2024-03-26T21:53:59.666521+0000","last_became_active":"2024-03-26T21:52:08.303240+0000","last_became_peered":"2024-03-26T21:52:08.303240+0000","last_unstale":"2024-03-26T21:53:59.666521+0000","last_undegraded":"2024-03-26T21:53:59.666521+0000","last_fullsized":"2024-03-26T21:53:59.666521+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:59.666446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:48.632355+0000","last_clean_scrub_stamp":"2024-03-26T21:53:59.666446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:00.275800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"364'14","reported_seq":507,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664872+0000","last_change":"2024-03-26T21:53:15.213177+0000","last_active":"2024-03-26T21:53:59.664872+0000","last_peered":"2024-03-26T21:53:59.664872+0000","last_clean":"2024-03-26T21:53:59.664872+0000","last_became_active":"2024-03-26T21:44:49.429047+0000","last_became_peered":"2024-03-26T21:44:49.429047+0000","last_unstale":"2024-03-26T21:53:59.664872+0000","last_undegraded":"2024-03-26T21:53:59.664872+0000","last_fullsized":"2024-03-26T21:53:59.664872+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'14","last_scrub_stamp":"2024-03-26T21:53:15.213078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.213078+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:20.643458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"534'49","reported_seq":804,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422270+0000","last_change":"2024-03-26T21:53:19.655516+0000","last_active":"2024-03-26T21:53:59.422270+0000","last_peered":"2024-03-26T21:53:59.422270+0000","last_clean":"2024-03-26T21:53:59.422270+0000","last_became_active":"2024-03-26T21:36:51.682502+0000","last_became_peered":"2024-03-26T21:36:51.682502+0000","last_unstale":"2024-03-26T21:53:59.422270+0000","last_undegraded":"2024-03-26T21:53:59.422270+0000","last_fullsized":"2024-03-26T21:53:59.422270+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'49","last_scrub_stamp":"2024-03-26T21:53:19.655382+0000","last_deep_scrub":"534'49","last_deep_scrub_stamp":"2024-03-26T21:53:19.655382+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.655382+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:44.067095+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"537'2","reported_seq":1055,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643568+0000","last_change":"2024-03-26T21:53:45.324633+0000","last_active":"2024-03-26T21:53:58.643568+0000","last_peered":"2024-03-26T21:53:58.643568+0000","last_clean":"2024-03-26T21:53:58.643568+0000","last_became_active":"2024-03-26T21:36:51.703212+0000","last_became_peered":"2024-03-26T21:36:51.703212+0000","last_unstale":"2024-03-26T21:53:58.643568+0000","last_undegraded":"2024-03-26T21:53:58.643568+0000","last_fullsized":"2024-03-26T21:53:58.643568+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'2","last_scrub_stamp":"2024-03-26T21:53:45.324543+0000","last_deep_scrub":"537'2","last_deep_scrub_stamp":"2024-03-26T21:53:45.324543+0000","last_clean_scrub_stamp":"2024-03-26T21:53:45.324543+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:05.741057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":964,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493150+0000","last_change":"2024-03-26T21:52:38.404963+0000","last_active":"2024-03-26T21:53:59.493150+0000","last_peered":"2024-03-26T21:53:59.493150+0000","last_clean":"2024-03-26T21:53:59.493150+0000","last_became_active":"2024-03-26T21:52:08.296363+0000","last_became_peered":"2024-03-26T21:52:08.296363+0000","last_unstale":"2024-03-26T21:53:59.493150+0000","last_undegraded":"2024-03-26T21:53:59.493150+0000","last_fullsized":"2024-03-26T21:53:59.493150+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.404842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.404842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:01.684210+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1346,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643192+0000","last_change":"2024-03-26T21:53:32.373011+0000","last_active":"2024-03-26T21:53:58.643192+0000","last_peered":"2024-03-26T21:53:58.643192+0000","last_clean":"2024-03-26T21:53:58.643192+0000","last_became_active":"2024-03-26T21:28:38.869421+0000","last_became_peered":"2024-03-26T21:28:38.869421+0000","last_unstale":"2024-03-26T21:53:58.643192+0000","last_undegraded":"2024-03-26T21:53:58.643192+0000","last_fullsized":"2024-03-26T21:53:58.643192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-26T21:53:32.372928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.372928+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.640659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'30","reported_seq":1137,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.426238+0000","last_change":"2024-03-26T21:53:35.317121+0000","last_active":"2024-03-26T21:53:59.426238+0000","last_peered":"2024-03-26T21:53:59.426238+0000","last_clean":"2024-03-26T21:53:59.426238+0000","last_became_active":"2024-03-26T21:36:51.711060+0000","last_became_peered":"2024-03-26T21:36:51.711060+0000","last_unstale":"2024-03-26T21:53:59.426238+0000","last_undegraded":"2024-03-26T21:53:59.426238+0000","last_fullsized":"2024-03-26T21:53:59.426238+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-26T21:53:35.317003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.317003+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.991721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1081,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769409+0000","last_change":"2024-03-26T21:52:30.766822+0000","last_active":"2024-03-26T21:53:59.769409+0000","last_peered":"2024-03-26T21:53:59.769409+0000","last_clean":"2024-03-26T21:53:59.769409+0000","last_became_active":"2024-03-26T21:52:08.294134+0000","last_became_peered":"2024-03-26T21:52:08.294134+0000","last_unstale":"2024-03-26T21:53:59.769409+0000","last_undegraded":"2024-03-26T21:53:59.769409+0000","last_fullsized":"2024-03-26T21:53:59.769409+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:30.766706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:32:35.525298+0000","last_clean_scrub_stamp":"2024-03-26T21:52:30.766706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1045,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642490+0000","last_change":"2024-03-26T21:52:58.505853+0000","last_active":"2024-03-26T21:53:58.642490+0000","last_peered":"2024-03-26T21:53:58.642490+0000","last_clean":"2024-03-26T21:53:58.642490+0000","last_became_active":"2024-03-26T21:52:08.301864+0000","last_became_peered":"2024-03-26T21:52:08.301864+0000","last_unstale":"2024-03-26T21:53:58.642490+0000","last_undegraded":"2024-03-26T21:53:58.642490+0000","last_fullsized":"2024-03-26T21:53:58.642490+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:58.505725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:58.505725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:19.925855+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":923,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822639+0000","last_change":"2024-03-26T21:53:19.361318+0000","last_active":"2024-03-26T21:53:59.822639+0000","last_peered":"2024-03-26T21:53:59.822639+0000","last_clean":"2024-03-26T21:53:59.822639+0000","last_became_active":"2024-03-26T21:53:02.593067+0000","last_became_peered":"2024-03-26T21:53:02.593067+0000","last_unstale":"2024-03-26T21:53:59.822639+0000","last_undegraded":"2024-03-26T21:53:59.822639+0000","last_fullsized":"2024-03-26T21:53:59.822639+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:19.361198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:33:24.229192+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.361198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.441219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"456'39","reported_seq":1104,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643300+0000","last_change":"2024-03-26T21:52:46.390309+0000","last_active":"2024-03-26T21:53:58.643300+0000","last_peered":"2024-03-26T21:53:58.643300+0000","last_clean":"2024-03-26T21:53:58.643300+0000","last_became_active":"2024-03-26T21:41:25.609719+0000","last_became_peered":"2024-03-26T21:41:25.609719+0000","last_unstale":"2024-03-26T21:53:58.643300+0000","last_undegraded":"2024-03-26T21:53:58.643300+0000","last_fullsized":"2024-03-26T21:53:58.643300+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:52:46.390213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:52:46.390213+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1079,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425742+0000","last_change":"2024-03-26T21:53:49.329111+0000","last_active":"2024-03-26T21:53:59.425742+0000","last_peered":"2024-03-26T21:53:59.425742+0000","last_clean":"2024-03-26T21:53:59.425742+0000","last_became_active":"2024-03-26T21:52:08.312067+0000","last_became_peered":"2024-03-26T21:52:08.312067+0000","last_unstale":"2024-03-26T21:53:59.425742+0000","last_undegraded":"2024-03-26T21:53:59.425742+0000","last_fullsized":"2024-03-26T21:53:59.425742+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:49.329008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:29:42.167719+0000","last_clean_scrub_stamp":"2024-03-26T21:53:49.329008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.054935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1051,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425266+0000","last_change":"2024-03-26T21:52:38.375617+0000","last_active":"2024-03-26T21:53:59.425266+0000","last_peered":"2024-03-26T21:53:59.425266+0000","last_clean":"2024-03-26T21:53:59.425266+0000","last_became_active":"2024-03-26T21:52:08.312506+0000","last_became_peered":"2024-03-26T21:52:08.312506+0000","last_unstale":"2024-03-26T21:53:59.425266+0000","last_undegraded":"2024-03-26T21:53:59.425266+0000","last_fullsized":"2024-03-26T21:53:59.425266+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.375489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:38.375489+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.375489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:07.708381+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1048,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769859+0000","last_change":"2024-03-26T21:53:38.811091+0000","last_active":"2024-03-26T21:53:59.769859+0000","last_peered":"2024-03-26T21:53:59.769859+0000","last_clean":"2024-03-26T21:53:59.769859+0000","last_became_active":"2024-03-26T21:28:38.869811+0000","last_became_peered":"2024-03-26T21:28:38.869811+0000","last_unstale":"2024-03-26T21:53:59.769859+0000","last_undegraded":"2024-03-26T21:53:59.769859+0000","last_fullsized":"2024-03-26T21:53:59.769859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:38.810965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:38.810965+0000","last_clean_scrub_stamp":"2024-03-26T21:53:38.810965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:45.765995+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"456'36","reported_seq":1150,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643547+0000","last_change":"2024-03-26T21:53:41.307161+0000","last_active":"2024-03-26T21:53:58.643547+0000","last_peered":"2024-03-26T21:53:58.643547+0000","last_clean":"2024-03-26T21:53:58.643547+0000","last_became_active":"2024-03-26T21:52:08.299332+0000","last_became_peered":"2024-03-26T21:52:08.299332+0000","last_unstale":"2024-03-26T21:53:58.643547+0000","last_undegraded":"2024-03-26T21:53:58.643547+0000","last_fullsized":"2024-03-26T21:53:58.643547+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'36","last_scrub_stamp":"2024-03-26T21:53:41.307050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.307050+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:55.228833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"245'6","reported_seq":2422,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:02.014381+0000","last_change":"2024-03-26T21:53:32.519457+0000","last_active":"2024-03-26T21:54:02.014381+0000","last_peered":"2024-03-26T21:54:02.014381+0000","last_clean":"2024-03-26T21:54:02.014381+0000","last_became_active":"2024-03-26T21:41:25.642081+0000","last_became_peered":"2024-03-26T21:41:25.642081+0000","last_unstale":"2024-03-26T21:54:02.014381+0000","last_undegraded":"2024-03-26T21:54:02.014381+0000","last_fullsized":"2024-03-26T21:54:02.014381+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'6","last_scrub_stamp":"2024-03-26T21:53:32.519338+0000","last_deep_scrub":"245'6","last_deep_scrub_stamp":"2024-03-26T21:53:32.519338+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.519338+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.021475+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046372999999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1407,"num_read_kb":1402,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1066,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643567+0000","last_change":"2024-03-26T21:52:54.317852+0000","last_active":"2024-03-26T21:53:58.643567+0000","last_peered":"2024-03-26T21:53:58.643567+0000","last_clean":"2024-03-26T21:53:58.643567+0000","last_became_active":"2024-03-26T21:44:49.438952+0000","last_became_peered":"2024-03-26T21:44:49.438952+0000","last_unstale":"2024-03-26T21:53:58.643567+0000","last_undegraded":"2024-03-26T21:53:58.643567+0000","last_fullsized":"2024-03-26T21:53:58.643567+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:54.317687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:47.735815+0000","last_clean_scrub_stamp":"2024-03-26T21:52:54.317687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:02.719799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1059,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425520+0000","last_change":"2024-03-26T21:53:12.442195+0000","last_active":"2024-03-26T21:53:59.425520+0000","last_peered":"2024-03-26T21:53:59.425520+0000","last_clean":"2024-03-26T21:53:59.425520+0000","last_became_active":"2024-03-26T21:44:49.442311+0000","last_became_peered":"2024-03-26T21:44:49.442311+0000","last_unstale":"2024-03-26T21:53:59.425520+0000","last_undegraded":"2024-03-26T21:53:59.425520+0000","last_fullsized":"2024-03-26T21:53:59.425520+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:12.442108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.442108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.535711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":989,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493278+0000","last_change":"2024-03-26T21:53:39.555788+0000","last_active":"2024-03-26T21:53:59.493278+0000","last_peered":"2024-03-26T21:53:59.493278+0000","last_clean":"2024-03-26T21:53:59.493278+0000","last_became_active":"2024-03-26T21:52:08.295236+0000","last_became_peered":"2024-03-26T21:52:08.295236+0000","last_unstale":"2024-03-26T21:53:59.493278+0000","last_undegraded":"2024-03-26T21:53:59.493278+0000","last_fullsized":"2024-03-26T21:53:59.493278+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:39.555685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.555685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.805657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"531'56","reported_seq":1320,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.823023+0000","last_change":"2024-03-26T21:53:02.594215+0000","last_active":"2024-03-26T21:53:59.823023+0000","last_peered":"2024-03-26T21:53:59.823023+0000","last_clean":"2024-03-26T21:53:59.823023+0000","last_became_active":"2024-03-26T21:53:02.593742+0000","last_became_peered":"2024-03-26T21:53:02.593742+0000","last_unstale":"2024-03-26T21:53:59.823023+0000","last_undegraded":"2024-03-26T21:53:59.823023+0000","last_fullsized":"2024-03-26T21:53:59.823023+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'55","last_scrub_stamp":"2024-03-26T21:52:45.761346+0000","last_deep_scrub":"468'55","last_deep_scrub_stamp":"2024-03-26T21:52:45.761346+0000","last_clean_scrub_stamp":"2024-03-26T21:52:45.761346+0000","objects_scrubbed":8,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.939882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":392,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664932+0000","last_change":"2024-03-26T21:53:33.185272+0000","last_active":"2024-03-26T21:53:59.664932+0000","last_peered":"2024-03-26T21:53:59.664932+0000","last_clean":"2024-03-26T21:53:59.664932+0000","last_became_active":"2024-03-26T21:44:49.400210+0000","last_became_peered":"2024-03-26T21:44:49.400210+0000","last_unstale":"2024-03-26T21:53:59.664932+0000","last_undegraded":"2024-03-26T21:53:59.664932+0000","last_fullsized":"2024-03-26T21:53:59.664932+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:33.185189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:41.689096+0000","last_clean_scrub_stamp":"2024-03-26T21:53:33.185189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.817991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1238,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492951+0000","last_change":"2024-03-26T21:52:52.366694+0000","last_active":"2024-03-26T21:53:59.492951+0000","last_peered":"2024-03-26T21:53:59.492951+0000","last_clean":"2024-03-26T21:53:59.492951+0000","last_became_active":"2024-03-26T21:41:25.642672+0000","last_became_peered":"2024-03-26T21:41:25.642672+0000","last_unstale":"2024-03-26T21:53:59.492951+0000","last_undegraded":"2024-03-26T21:53:59.492951+0000","last_fullsized":"2024-03-26T21:53:59.492951+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-26T21:52:52.366588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:26:14.853821+0000","last_clean_scrub_stamp":"2024-03-26T21:52:52.366588+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.435534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1069,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769679+0000","last_change":"2024-03-26T21:52:38.814451+0000","last_active":"2024-03-26T21:53:59.769679+0000","last_peered":"2024-03-26T21:53:59.769679+0000","last_clean":"2024-03-26T21:53:59.769679+0000","last_became_active":"2024-03-26T21:44:49.334453+0000","last_became_peered":"2024-03-26T21:44:49.334453+0000","last_unstale":"2024-03-26T21:53:59.769679+0000","last_undegraded":"2024-03-26T21:53:59.769679+0000","last_fullsized":"2024-03-26T21:53:59.769679+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:38.814353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:38.814353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":99,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424862+0000","last_change":"2024-03-26T21:53:29.782827+0000","last_active":"2024-03-26T21:53:59.424862+0000","last_peered":"2024-03-26T21:53:59.424862+0000","last_clean":"2024-03-26T21:53:59.424862+0000","last_became_active":"2024-03-26T21:52:08.296077+0000","last_became_peered":"2024-03-26T21:52:08.296077+0000","last_unstale":"2024-03-26T21:53:59.424862+0000","last_undegraded":"2024-03-26T21:53:59.424862+0000","last_fullsized":"2024-03-26T21:53:59.424862+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:29.782734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:32.868200+0000","last_clean_scrub_stamp":"2024-03-26T21:53:29.782734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.219050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1098,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425750+0000","last_change":"2024-03-26T21:52:42.368231+0000","last_active":"2024-03-26T21:53:59.425750+0000","last_peered":"2024-03-26T21:53:59.425750+0000","last_clean":"2024-03-26T21:53:59.425750+0000","last_became_active":"2024-03-26T21:28:36.841743+0000","last_became_peered":"2024-03-26T21:28:36.841743+0000","last_unstale":"2024-03-26T21:53:59.425750+0000","last_undegraded":"2024-03-26T21:53:59.425750+0000","last_fullsized":"2024-03-26T21:53:59.425750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-26T21:52:42.368124+0000","last_deep_scrub":"88'18","last_deep_scrub_stamp":"2024-03-26T21:29:55.054601+0000","last_clean_scrub_stamp":"2024-03-26T21:52:42.368124+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:05.954104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424436+0000","last_change":"2024-03-26T21:53:34.784060+0000","last_active":"2024-03-26T21:53:59.424436+0000","last_peered":"2024-03-26T21:53:59.424436+0000","last_clean":"2024-03-26T21:53:59.424436+0000","last_became_active":"2024-03-26T21:52:08.303990+0000","last_became_peered":"2024-03-26T21:52:08.303990+0000","last_unstale":"2024-03-26T21:53:59.424436+0000","last_undegraded":"2024-03-26T21:53:59.424436+0000","last_fullsized":"2024-03-26T21:53:59.424436+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.783951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:34.783951+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.783951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:37.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001090024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1079,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425376+0000","last_change":"2024-03-26T21:53:07.427049+0000","last_active":"2024-03-26T21:53:59.425376+0000","last_peered":"2024-03-26T21:53:59.425376+0000","last_clean":"2024-03-26T21:53:59.425376+0000","last_became_active":"2024-03-26T21:52:08.311751+0000","last_became_peered":"2024-03-26T21:52:08.311751+0000","last_unstale":"2024-03-26T21:53:59.425376+0000","last_undegraded":"2024-03-26T21:53:59.425376+0000","last_fullsized":"2024-03-26T21:53:59.425376+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:07.426936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:46:22.650296+0000","last_clean_scrub_stamp":"2024-03-26T21:53:07.426936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.805222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":957,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493313+0000","last_change":"2024-03-26T21:53:01.495558+0000","last_active":"2024-03-26T21:53:59.493313+0000","last_peered":"2024-03-26T21:53:59.493313+0000","last_clean":"2024-03-26T21:53:59.493313+0000","last_became_active":"2024-03-26T21:44:49.394652+0000","last_became_peered":"2024-03-26T21:44:49.394652+0000","last_unstale":"2024-03-26T21:53:59.493313+0000","last_undegraded":"2024-03-26T21:53:59.493313+0000","last_fullsized":"2024-03-26T21:53:59.493313+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.495451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:19.718867+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.495451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.142605+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642915+0000","last_change":"2024-03-26T21:52:36.359809+0000","last_active":"2024-03-26T21:53:58.642915+0000","last_peered":"2024-03-26T21:53:58.642915+0000","last_clean":"2024-03-26T21:53:58.642915+0000","last_became_active":"2024-03-26T21:36:51.721617+0000","last_became_peered":"2024-03-26T21:36:51.721617+0000","last_unstale":"2024-03-26T21:53:58.642915+0000","last_undegraded":"2024-03-26T21:53:58.642915+0000","last_fullsized":"2024-03-26T21:53:58.642915+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:36.359706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:32.000079+0000","last_clean_scrub_stamp":"2024-03-26T21:52:36.359706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:58.945690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'67","reported_seq":1521,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:00.193066+0000","last_change":"2024-03-26T21:53:59.771072+0000","last_active":"2024-03-26T21:54:00.193066+0000","last_peered":"2024-03-26T21:54:00.193066+0000","last_clean":"2024-03-26T21:54:00.193066+0000","last_became_active":"2024-03-26T21:52:08.299794+0000","last_became_peered":"2024-03-26T21:52:08.299794+0000","last_unstale":"2024-03-26T21:54:00.193066+0000","last_undegraded":"2024-03-26T21:54:00.193066+0000","last_fullsized":"2024-03-26T21:54:00.193066+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'67","last_scrub_stamp":"2024-03-26T21:53:59.770944+0000","last_deep_scrub":"513'67","last_deep_scrub_stamp":"2024-03-26T21:52:28.733406+0000","last_clean_scrub_stamp":"2024-03-26T21:53:59.770944+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:27.302685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422223+0000","last_change":"2024-03-26T21:53:54.702335+0000","last_active":"2024-03-26T21:53:59.422223+0000","last_peered":"2024-03-26T21:53:59.422223+0000","last_clean":"2024-03-26T21:53:59.422223+0000","last_became_active":"2024-03-26T21:48:58.975703+0000","last_became_peered":"2024-03-26T21:48:58.975703+0000","last_unstale":"2024-03-26T21:53:59.422223+0000","last_undegraded":"2024-03-26T21:53:59.422223+0000","last_fullsized":"2024-03-26T21:53:59.422223+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:54.702226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:41.701840+0000","last_clean_scrub_stamp":"2024-03-26T21:53:54.702226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:21.217308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"235'3","reported_seq":1067,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.768904+0000","last_change":"2024-03-26T21:52:32.770619+0000","last_active":"2024-03-26T21:53:59.768904+0000","last_peered":"2024-03-26T21:53:59.768904+0000","last_clean":"2024-03-26T21:53:59.768904+0000","last_became_active":"2024-03-26T21:28:34.847825+0000","last_became_peered":"2024-03-26T21:28:34.847825+0000","last_unstale":"2024-03-26T21:53:59.768904+0000","last_undegraded":"2024-03-26T21:53:59.768904+0000","last_fullsized":"2024-03-26T21:53:59.768904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'3","last_scrub_stamp":"2024-03-26T21:52:32.770524+0000","last_deep_scrub":"235'3","last_deep_scrub_stamp":"2024-03-26T21:48:43.783585+0000","last_clean_scrub_stamp":"2024-03-26T21:52:32.770524+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013284480000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":384,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665396+0000","last_change":"2024-03-26T21:52:33.083793+0000","last_active":"2024-03-26T21:53:59.665396+0000","last_peered":"2024-03-26T21:53:59.665396+0000","last_clean":"2024-03-26T21:53:59.665396+0000","last_became_active":"2024-03-26T21:44:49.401753+0000","last_became_peered":"2024-03-26T21:44:49.401753+0000","last_unstale":"2024-03-26T21:53:59.665396+0000","last_undegraded":"2024-03-26T21:53:59.665396+0000","last_fullsized":"2024-03-26T21:53:59.665396+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:33.083710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:33.083710+0000","last_clean_scrub_stamp":"2024-03-26T21:52:33.083710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1111,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643422+0000","last_change":"2024-03-26T21:53:21.491076+0000","last_active":"2024-03-26T21:53:58.643422+0000","last_peered":"2024-03-26T21:53:58.643422+0000","last_clean":"2024-03-26T21:53:58.643422+0000","last_became_active":"2024-03-26T21:52:17.976074+0000","last_became_peered":"2024-03-26T21:52:17.976074+0000","last_unstale":"2024-03-26T21:53:58.643422+0000","last_undegraded":"2024-03-26T21:53:58.643422+0000","last_fullsized":"2024-03-26T21:53:58.643422+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-26T21:53:21.490925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:21.490925+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:36.818825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1076,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769393+0000","last_change":"2024-03-26T21:53:11.707647+0000","last_active":"2024-03-26T21:53:59.769393+0000","last_peered":"2024-03-26T21:53:59.769393+0000","last_clean":"2024-03-26T21:53:59.769393+0000","last_became_active":"2024-03-26T21:44:49.334168+0000","last_became_peered":"2024-03-26T21:44:49.334168+0000","last_unstale":"2024-03-26T21:53:59.769393+0000","last_undegraded":"2024-03-26T21:53:59.769393+0000","last_fullsized":"2024-03-26T21:53:59.769393+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:11.707547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:39.585631+0000","last_clean_scrub_stamp":"2024-03-26T21:53:11.707547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:23.254548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":99,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424624+0000","last_change":"2024-03-26T21:53:50.795495+0000","last_active":"2024-03-26T21:53:59.424624+0000","last_peered":"2024-03-26T21:53:59.424624+0000","last_clean":"2024-03-26T21:53:59.424624+0000","last_became_active":"2024-03-26T21:52:08.297815+0000","last_became_peered":"2024-03-26T21:52:08.297815+0000","last_unstale":"2024-03-26T21:53:59.424624+0000","last_undegraded":"2024-03-26T21:53:59.424624+0000","last_fullsized":"2024-03-26T21:53:59.424624+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:50.795415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:50.795415+0000","last_clean_scrub_stamp":"2024-03-26T21:53:50.795415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:15.674635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1041,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643708+0000","last_change":"2024-03-26T21:53:30.480975+0000","last_active":"2024-03-26T21:53:58.643708+0000","last_peered":"2024-03-26T21:53:58.643708+0000","last_clean":"2024-03-26T21:53:58.643708+0000","last_became_active":"2024-03-26T21:52:08.299836+0000","last_became_peered":"2024-03-26T21:52:08.299836+0000","last_unstale":"2024-03-26T21:53:58.643708+0000","last_undegraded":"2024-03-26T21:53:58.643708+0000","last_fullsized":"2024-03-26T21:53:58.643708+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:30.480878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:31:37.721821+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.480878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.420091+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1024,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822637+0000","last_change":"2024-03-26T21:53:26.261237+0000","last_active":"2024-03-26T21:53:59.822637+0000","last_peered":"2024-03-26T21:53:59.822637+0000","last_clean":"2024-03-26T21:53:59.822637+0000","last_became_active":"2024-03-26T21:53:02.592322+0000","last_became_peered":"2024-03-26T21:53:02.592322+0000","last_unstale":"2024-03-26T21:53:59.822637+0000","last_undegraded":"2024-03-26T21:53:59.822637+0000","last_fullsized":"2024-03-26T21:53:59.822637+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'60","last_scrub_stamp":"2024-03-26T21:53:26.261097+0000","last_deep_scrub":"110'28","last_deep_scrub_stamp":"2024-03-26T21:33:25.213564+0000","last_clean_scrub_stamp":"2024-03-26T21:53:26.261097+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:33.939070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":999,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:03.575819+0000","last_change":"2024-03-26T21:54:03.575839+0000","last_active":"2024-03-26T21:54:03.575819+0000","last_peered":"2024-03-26T21:54:03.575819+0000","last_clean":"2024-03-26T21:54:03.575819+0000","last_became_active":"2024-03-26T21:41:25.626354+0000","last_became_peered":"2024-03-26T21:41:25.626354+0000","last_unstale":"2024-03-26T21:54:03.575819+0000","last_undegraded":"2024-03-26T21:54:03.575819+0000","last_fullsized":"2024-03-26T21:54:03.575819+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:54:03.575755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:54:03.575755+0000","last_clean_scrub_stamp":"2024-03-26T21:54:03.575755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:12.097110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":91,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424564+0000","last_change":"2024-03-26T21:53:16.778514+0000","last_active":"2024-03-26T21:53:59.424564+0000","last_peered":"2024-03-26T21:53:59.424564+0000","last_clean":"2024-03-26T21:53:59.424564+0000","last_became_active":"2024-03-26T21:52:08.303310+0000","last_became_peered":"2024-03-26T21:52:08.303310+0000","last_unstale":"2024-03-26T21:53:59.424564+0000","last_undegraded":"2024-03-26T21:53:59.424564+0000","last_fullsized":"2024-03-26T21:53:59.424564+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:16.778380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:02.460651+0000","last_clean_scrub_stamp":"2024-03-26T21:53:16.778380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.517410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":976,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492582+0000","last_change":"2024-03-26T21:53:13.668492+0000","last_active":"2024-03-26T21:53:59.492582+0000","last_peered":"2024-03-26T21:53:59.492582+0000","last_clean":"2024-03-26T21:53:59.492582+0000","last_became_active":"2024-03-26T21:52:08.299894+0000","last_became_peered":"2024-03-26T21:52:08.299894+0000","last_unstale":"2024-03-26T21:53:59.492582+0000","last_undegraded":"2024-03-26T21:53:59.492582+0000","last_fullsized":"2024-03-26T21:53:59.492582+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:13.668408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:13.668408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:38.983061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":99,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424282+0000","last_change":"2024-03-26T21:53:24.771160+0000","last_active":"2024-03-26T21:53:59.424282+0000","last_peered":"2024-03-26T21:53:59.424282+0000","last_clean":"2024-03-26T21:53:59.424282+0000","last_became_active":"2024-03-26T21:52:08.304374+0000","last_became_peered":"2024-03-26T21:52:08.304374+0000","last_unstale":"2024-03-26T21:53:59.424282+0000","last_undegraded":"2024-03-26T21:53:59.424282+0000","last_fullsized":"2024-03-26T21:53:59.424282+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.771068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:54.268871+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.771068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.161836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":819,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422324+0000","last_change":"2024-03-26T21:53:10.746777+0000","last_active":"2024-03-26T21:53:59.422324+0000","last_peered":"2024-03-26T21:53:59.422324+0000","last_clean":"2024-03-26T21:53:59.422324+0000","last_became_active":"2024-03-26T21:52:17.977990+0000","last_became_peered":"2024-03-26T21:52:17.977990+0000","last_unstale":"2024-03-26T21:53:59.422324+0000","last_undegraded":"2024-03-26T21:53:59.422324+0000","last_fullsized":"2024-03-26T21:53:59.422324+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-26T21:53:10.746662+0000","last_deep_scrub":"282'23","last_deep_scrub_stamp":"2024-03-26T21:48:36.706278+0000","last_clean_scrub_stamp":"2024-03-26T21:53:10.746662+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:24.942562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"88'2","reported_seq":1021,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493590+0000","last_change":"2024-03-26T21:52:48.378422+0000","last_active":"2024-03-26T21:53:59.493590+0000","last_peered":"2024-03-26T21:53:59.493590+0000","last_clean":"2024-03-26T21:53:59.493590+0000","last_became_active":"2024-03-26T21:41:25.610230+0000","last_became_peered":"2024-03-26T21:41:25.610230+0000","last_unstale":"2024-03-26T21:53:59.493590+0000","last_undegraded":"2024-03-26T21:53:59.493590+0000","last_fullsized":"2024-03-26T21:53:59.493590+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T21:52:48.378250+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T21:48:35.727841+0000","last_clean_scrub_stamp":"2024-03-26T21:52:48.378250+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:04.429955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":392,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665118+0000","last_change":"2024-03-26T21:53:34.344239+0000","last_active":"2024-03-26T21:53:59.665118+0000","last_peered":"2024-03-26T21:53:59.665118+0000","last_clean":"2024-03-26T21:53:59.665118+0000","last_became_active":"2024-03-26T21:44:49.424964+0000","last_became_peered":"2024-03-26T21:44:49.424964+0000","last_unstale":"2024-03-26T21:53:59.665118+0000","last_undegraded":"2024-03-26T21:53:59.665118+0000","last_fullsized":"2024-03-26T21:53:59.665118+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:34.344127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:34.344127+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.344127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:59.284054+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"534'15","reported_seq":1291,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492759+0000","last_change":"2024-03-26T21:53:15.609556+0000","last_active":"2024-03-26T21:53:59.492759+0000","last_peered":"2024-03-26T21:53:59.492759+0000","last_clean":"2024-03-26T21:53:59.492759+0000","last_became_active":"2024-03-26T21:41:25.623353+0000","last_became_peered":"2024-03-26T21:41:25.623353+0000","last_unstale":"2024-03-26T21:53:59.492759+0000","last_undegraded":"2024-03-26T21:53:59.492759+0000","last_fullsized":"2024-03-26T21:53:59.492759+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'15","last_scrub_stamp":"2024-03-26T21:53:15.609420+0000","last_deep_scrub":"534'15","last_deep_scrub_stamp":"2024-03-26T21:53:15.609420+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.609420+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.962929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"469'46","reported_seq":1129,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.768678+0000","last_change":"2024-03-26T21:53:06.764246+0000","last_active":"2024-03-26T21:53:59.768678+0000","last_peered":"2024-03-26T21:53:59.768678+0000","last_clean":"2024-03-26T21:53:59.768678+0000","last_became_active":"2024-03-26T21:41:25.624645+0000","last_became_peered":"2024-03-26T21:41:25.624645+0000","last_unstale":"2024-03-26T21:53:59.768678+0000","last_undegraded":"2024-03-26T21:53:59.768678+0000","last_fullsized":"2024-03-26T21:53:59.768678+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'46","last_scrub_stamp":"2024-03-26T21:53:06.764090+0000","last_deep_scrub":"469'46","last_deep_scrub_stamp":"2024-03-26T21:49:38.645456+0000","last_clean_scrub_stamp":"2024-03-26T21:53:06.764090+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.496953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'3","reported_seq":1038,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493577+0000","last_change":"2024-03-26T21:53:50.604486+0000","last_active":"2024-03-26T21:53:59.493577+0000","last_peered":"2024-03-26T21:53:59.493577+0000","last_clean":"2024-03-26T21:53:59.493577+0000","last_became_active":"2024-03-26T21:41:25.610845+0000","last_became_peered":"2024-03-26T21:41:25.610845+0000","last_unstale":"2024-03-26T21:53:59.493577+0000","last_undegraded":"2024-03-26T21:53:59.493577+0000","last_fullsized":"2024-03-26T21:53:59.493577+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'3","last_scrub_stamp":"2024-03-26T21:53:50.604363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:50.604363+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.940740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1044,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769146+0000","last_change":"2024-03-26T21:53:07.773451+0000","last_active":"2024-03-26T21:53:59.769146+0000","last_peered":"2024-03-26T21:53:59.769146+0000","last_clean":"2024-03-26T21:53:59.769146+0000","last_became_active":"2024-03-26T21:28:41.113807+0000","last_became_peered":"2024-03-26T21:28:41.113807+0000","last_unstale":"2024-03-26T21:53:59.769146+0000","last_undegraded":"2024-03-26T21:53:59.769146+0000","last_fullsized":"2024-03-26T21:53:59.769146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:07.773327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:07.773327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:08.760526+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"456'18","reported_seq":422,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665596+0000","last_change":"2024-03-26T21:53:51.090602+0000","last_active":"2024-03-26T21:53:59.665596+0000","last_peered":"2024-03-26T21:53:59.665596+0000","last_clean":"2024-03-26T21:53:59.665596+0000","last_became_active":"2024-03-26T21:48:58.989898+0000","last_became_peered":"2024-03-26T21:48:58.989898+0000","last_unstale":"2024-03-26T21:53:59.665596+0000","last_undegraded":"2024-03-26T21:53:59.665596+0000","last_fullsized":"2024-03-26T21:53:59.665596+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'18","last_scrub_stamp":"2024-03-26T21:53:51.090514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:51.090514+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.695920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":930,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822193+0000","last_change":"2024-03-26T21:53:02.594560+0000","last_active":"2024-03-26T21:53:59.822193+0000","last_peered":"2024-03-26T21:53:59.822193+0000","last_clean":"2024-03-26T21:53:59.822193+0000","last_became_active":"2024-03-26T21:53:02.594195+0000","last_became_peered":"2024-03-26T21:53:02.594195+0000","last_unstale":"2024-03-26T21:53:59.822193+0000","last_undegraded":"2024-03-26T21:53:59.822193+0000","last_fullsized":"2024-03-26T21:53:59.822193+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.380797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:37.291965+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.380797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:21.247952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1059,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769553+0000","last_change":"2024-03-26T21:52:50.710856+0000","last_active":"2024-03-26T21:53:59.769553+0000","last_peered":"2024-03-26T21:53:59.769553+0000","last_clean":"2024-03-26T21:53:59.769553+0000","last_became_active":"2024-03-26T21:44:49.398613+0000","last_became_peered":"2024-03-26T21:44:49.398613+0000","last_unstale":"2024-03-26T21:53:59.769553+0000","last_undegraded":"2024-03-26T21:53:59.769553+0000","last_fullsized":"2024-03-26T21:53:59.769553+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:50.710777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.710777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:53:59.799184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":974,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493316+0000","last_change":"2024-03-26T21:53:35.611244+0000","last_active":"2024-03-26T21:53:59.493316+0000","last_peered":"2024-03-26T21:53:59.493316+0000","last_clean":"2024-03-26T21:53:59.493316+0000","last_became_active":"2024-03-26T21:41:25.621861+0000","last_became_peered":"2024-03-26T21:41:25.621861+0000","last_unstale":"2024-03-26T21:53:59.493316+0000","last_undegraded":"2024-03-26T21:53:59.493316+0000","last_fullsized":"2024-03-26T21:53:59.493316+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.611124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:22.364735+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.611124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:39.399290+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1104,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643674+0000","last_change":"2024-03-26T21:53:04.374991+0000","last_active":"2024-03-26T21:53:58.643674+0000","last_peered":"2024-03-26T21:53:58.643674+0000","last_clean":"2024-03-26T21:53:58.643674+0000","last_became_active":"2024-03-26T21:44:49.441074+0000","last_became_peered":"2024-03-26T21:44:49.441074+0000","last_unstale":"2024-03-26T21:53:58.643674+0000","last_undegraded":"2024-03-26T21:53:58.643674+0000","last_fullsized":"2024-03-26T21:53:58.643674+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:04.374896+0000","last_deep_scrub":"113'13","last_deep_scrub_stamp":"2024-03-26T21:31:38.685453+0000","last_clean_scrub_stamp":"2024-03-26T21:53:04.374896+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.011033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1011,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492861+0000","last_change":"2024-03-26T21:53:14.653735+0000","last_active":"2024-03-26T21:53:59.492861+0000","last_peered":"2024-03-26T21:53:59.492861+0000","last_clean":"2024-03-26T21:53:59.492861+0000","last_became_active":"2024-03-26T21:41:25.622837+0000","last_became_peered":"2024-03-26T21:41:25.622837+0000","last_unstale":"2024-03-26T21:53:59.492861+0000","last_undegraded":"2024-03-26T21:53:59.492861+0000","last_fullsized":"2024-03-26T21:53:59.492861+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:14.653627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:14.653627+0000","last_clean_scrub_stamp":"2024-03-26T21:53:14.653627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:32.483533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":702,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425584+0000","last_change":"2024-03-26T21:53:15.374854+0000","last_active":"2024-03-26T21:53:59.425584+0000","last_peered":"2024-03-26T21:53:59.425584+0000","last_clean":"2024-03-26T21:53:59.425584+0000","last_became_active":"2024-03-26T21:52:08.312931+0000","last_became_peered":"2024-03-26T21:52:08.312931+0000","last_unstale":"2024-03-26T21:53:59.425584+0000","last_undegraded":"2024-03-26T21:53:59.425584+0000","last_fullsized":"2024-03-26T21:53:59.425584+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:15.374770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:15.374770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:20.037797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":91,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424839+0000","last_change":"2024-03-26T21:53:03.653511+0000","last_active":"2024-03-26T21:53:59.424839+0000","last_peered":"2024-03-26T21:53:59.424839+0000","last_clean":"2024-03-26T21:53:59.424839+0000","last_became_active":"2024-03-26T21:52:08.295874+0000","last_became_peered":"2024-03-26T21:52:08.295874+0000","last_unstale":"2024-03-26T21:53:59.424839+0000","last_undegraded":"2024-03-26T21:53:59.424839+0000","last_fullsized":"2024-03-26T21:53:59.424839+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.653366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.653366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:22.129353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":1128,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769632+0000","last_change":"2024-03-26T21:53:41.819511+0000","last_active":"2024-03-26T21:53:59.769632+0000","last_peered":"2024-03-26T21:53:59.769632+0000","last_clean":"2024-03-26T21:53:59.769632+0000","last_became_active":"2024-03-26T21:44:49.334581+0000","last_became_peered":"2024-03-26T21:44:49.334581+0000","last_unstale":"2024-03-26T21:53:59.769632+0000","last_undegraded":"2024-03-26T21:53:59.769632+0000","last_fullsized":"2024-03-26T21:53:59.769632+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:41.819382+0000","last_deep_scrub":"146'13","last_deep_scrub_stamp":"2024-03-26T21:33:29.776392+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.819382+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.581578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642887+0000","last_change":"2024-03-26T21:53:05.346212+0000","last_active":"2024-03-26T21:53:58.642887+0000","last_peered":"2024-03-26T21:53:58.642887+0000","last_clean":"2024-03-26T21:53:58.642887+0000","last_became_active":"2024-03-26T21:44:49.440432+0000","last_became_peered":"2024-03-26T21:44:49.440432+0000","last_unstale":"2024-03-26T21:53:58.642887+0000","last_undegraded":"2024-03-26T21:53:58.642887+0000","last_fullsized":"2024-03-26T21:53:58.642887+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:05.346112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:05.346112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:10.311996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"505'15","reported_seq":519,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665341+0000","last_change":"2024-03-26T21:52:44.140183+0000","last_active":"2024-03-26T21:53:59.665341+0000","last_peered":"2024-03-26T21:53:59.665341+0000","last_clean":"2024-03-26T21:53:59.665341+0000","last_became_active":"2024-03-26T21:52:08.303917+0000","last_became_peered":"2024-03-26T21:52:08.303917+0000","last_unstale":"2024-03-26T21:53:59.665341+0000","last_undegraded":"2024-03-26T21:53:59.665341+0000","last_fullsized":"2024-03-26T21:53:59.665341+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'15","last_scrub_stamp":"2024-03-26T21:52:44.140065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:44.140065+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":749,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423308+0000","last_change":"2024-03-26T21:53:32.760301+0000","last_active":"2024-03-26T21:53:59.423308+0000","last_peered":"2024-03-26T21:53:59.423308+0000","last_clean":"2024-03-26T21:53:59.423308+0000","last_became_active":"2024-03-26T21:52:08.295768+0000","last_became_peered":"2024-03-26T21:52:08.295768+0000","last_unstale":"2024-03-26T21:53:59.423308+0000","last_undegraded":"2024-03-26T21:53:59.423308+0000","last_fullsized":"2024-03-26T21:53:59.423308+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:32.760194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:02.948417+0000","last_clean_scrub_stamp":"2024-03-26T21:53:32.760194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:59.759274+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"456'35","reported_seq":418,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665095+0000","last_change":"2024-03-26T21:53:22.301307+0000","last_active":"2024-03-26T21:53:59.665095+0000","last_peered":"2024-03-26T21:53:59.665095+0000","last_clean":"2024-03-26T21:53:59.665095+0000","last_became_active":"2024-03-26T21:44:49.431947+0000","last_became_peered":"2024-03-26T21:44:49.431947+0000","last_unstale":"2024-03-26T21:53:59.665095+0000","last_undegraded":"2024-03-26T21:53:59.665095+0000","last_fullsized":"2024-03-26T21:53:59.665095+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-26T21:53:22.301180+0000","last_deep_scrub":"166'15","last_deep_scrub_stamp":"2024-03-26T21:34:27.403785+0000","last_clean_scrub_stamp":"2024-03-26T21:53:22.301180+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.738037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":996,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493308+0000","last_change":"2024-03-26T21:53:03.498814+0000","last_active":"2024-03-26T21:53:59.493308+0000","last_peered":"2024-03-26T21:53:59.493308+0000","last_clean":"2024-03-26T21:53:59.493308+0000","last_became_active":"2024-03-26T21:52:08.305000+0000","last_became_peered":"2024-03-26T21:52:08.305000+0000","last_unstale":"2024-03-26T21:53:59.493308+0000","last_undegraded":"2024-03-26T21:53:59.493308+0000","last_fullsized":"2024-03-26T21:53:59.493308+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.498714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.498714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:04.371324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"88'9","reported_seq":1354,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769400+0000","last_change":"2024-03-26T21:52:26.747492+0000","last_active":"2024-03-26T21:53:59.769400+0000","last_peered":"2024-03-26T21:53:59.769400+0000","last_clean":"2024-03-26T21:53:59.769400+0000","last_became_active":"2024-03-26T21:28:38.866350+0000","last_became_peered":"2024-03-26T21:28:38.866350+0000","last_unstale":"2024-03-26T21:53:59.769400+0000","last_undegraded":"2024-03-26T21:53:59.769400+0000","last_fullsized":"2024-03-26T21:53:59.769400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'9","last_scrub_stamp":"2024-03-26T21:52:26.747371+0000","last_deep_scrub":"88'9","last_deep_scrub_stamp":"2024-03-26T21:52:26.747371+0000","last_clean_scrub_stamp":"2024-03-26T21:52:26.747371+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":1047,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643461+0000","last_change":"2024-03-26T21:53:12.461418+0000","last_active":"2024-03-26T21:53:58.643461+0000","last_peered":"2024-03-26T21:53:58.643461+0000","last_clean":"2024-03-26T21:53:58.643461+0000","last_became_active":"2024-03-26T21:44:49.440563+0000","last_became_peered":"2024-03-26T21:44:49.440563+0000","last_unstale":"2024-03-26T21:53:58.643461+0000","last_undegraded":"2024-03-26T21:53:58.643461+0000","last_fullsized":"2024-03-26T21:53:58.643461+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:53:12.461268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:12.461268+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.000302+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":779,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422546+0000","last_change":"2024-03-26T21:53:27.667095+0000","last_active":"2024-03-26T21:53:59.422546+0000","last_peered":"2024-03-26T21:53:59.422546+0000","last_clean":"2024-03-26T21:53:59.422546+0000","last_became_active":"2024-03-26T21:36:51.700281+0000","last_became_peered":"2024-03-26T21:36:51.700281+0000","last_unstale":"2024-03-26T21:53:59.422546+0000","last_undegraded":"2024-03-26T21:53:59.422546+0000","last_fullsized":"2024-03-26T21:53:59.422546+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:27.666957+0000","last_deep_scrub":"424'24","last_deep_scrub_stamp":"2024-03-26T21:47:22.503147+0000","last_clean_scrub_stamp":"2024-03-26T21:53:27.666957+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.271390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":739,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422609+0000","last_change":"2024-03-26T21:52:54.729667+0000","last_active":"2024-03-26T21:53:59.422609+0000","last_peered":"2024-03-26T21:53:59.422609+0000","last_clean":"2024-03-26T21:53:59.422609+0000","last_became_active":"2024-03-26T21:41:25.601731+0000","last_became_peered":"2024-03-26T21:41:25.601731+0000","last_unstale":"2024-03-26T21:53:59.422609+0000","last_undegraded":"2024-03-26T21:53:59.422609+0000","last_fullsized":"2024-03-26T21:53:59.422609+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:54.729584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:39.735146+0000","last_clean_scrub_stamp":"2024-03-26T21:52:54.729584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:09.427352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1073,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.426188+0000","last_change":"2024-03-26T21:53:48.325613+0000","last_active":"2024-03-26T21:53:59.426188+0000","last_peered":"2024-03-26T21:53:59.426188+0000","last_clean":"2024-03-26T21:53:59.426188+0000","last_became_active":"2024-03-26T21:44:49.397696+0000","last_became_peered":"2024-03-26T21:44:49.397696+0000","last_unstale":"2024-03-26T21:53:59.426188+0000","last_undegraded":"2024-03-26T21:53:59.426188+0000","last_fullsized":"2024-03-26T21:53:59.426188+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:48.325518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:24.700160+0000","last_clean_scrub_stamp":"2024-03-26T21:53:48.325518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:52.642317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":391,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664740+0000","last_change":"2024-03-26T21:53:31.214121+0000","last_active":"2024-03-26T21:53:59.664740+0000","last_peered":"2024-03-26T21:53:59.664740+0000","last_clean":"2024-03-26T21:53:59.664740+0000","last_became_active":"2024-03-26T21:44:49.407321+0000","last_became_peered":"2024-03-26T21:44:49.407321+0000","last_unstale":"2024-03-26T21:53:59.664740+0000","last_undegraded":"2024-03-26T21:53:59.664740+0000","last_fullsized":"2024-03-26T21:53:59.664740+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:31.214018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:40:49.810420+0000","last_clean_scrub_stamp":"2024-03-26T21:53:31.214018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.213833+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"456'25","reported_seq":1123,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.426286+0000","last_change":"2024-03-26T21:53:34.337707+0000","last_active":"2024-03-26T21:53:59.426286+0000","last_peered":"2024-03-26T21:53:59.426286+0000","last_clean":"2024-03-26T21:53:59.426286+0000","last_became_active":"2024-03-26T21:52:08.312344+0000","last_became_peered":"2024-03-26T21:52:08.312344+0000","last_unstale":"2024-03-26T21:53:59.426286+0000","last_undegraded":"2024-03-26T21:53:59.426286+0000","last_fullsized":"2024-03-26T21:53:59.426286+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:34.337568+0000","last_deep_scrub":"91'13","last_deep_scrub_stamp":"2024-03-26T21:37:43.412986+0000","last_clean_scrub_stamp":"2024-03-26T21:53:34.337568+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.277211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":989,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493491+0000","last_change":"2024-03-26T21:53:23.699793+0000","last_active":"2024-03-26T21:53:59.493491+0000","last_peered":"2024-03-26T21:53:59.493491+0000","last_clean":"2024-03-26T21:53:59.493491+0000","last_became_active":"2024-03-26T21:44:49.395152+0000","last_became_peered":"2024-03-26T21:44:49.395152+0000","last_unstale":"2024-03-26T21:53:59.493491+0000","last_undegraded":"2024-03-26T21:53:59.493491+0000","last_fullsized":"2024-03-26T21:53:59.493491+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:23.699681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:23.699681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:44.150085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":383,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664957+0000","last_change":"2024-03-26T21:52:23.047307+0000","last_active":"2024-03-26T21:53:59.664957+0000","last_peered":"2024-03-26T21:53:59.664957+0000","last_clean":"2024-03-26T21:53:59.664957+0000","last_became_active":"2024-03-26T21:44:49.434588+0000","last_became_peered":"2024-03-26T21:44:49.434588+0000","last_unstale":"2024-03-26T21:53:59.664957+0000","last_undegraded":"2024-03-26T21:53:59.664957+0000","last_fullsized":"2024-03-26T21:53:59.664957+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:23.047219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:23.047219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1027,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642359+0000","last_change":"2024-03-26T21:53:35.383088+0000","last_active":"2024-03-26T21:53:58.642359+0000","last_peered":"2024-03-26T21:53:58.642359+0000","last_clean":"2024-03-26T21:53:58.642359+0000","last_became_active":"2024-03-26T21:28:40.888796+0000","last_became_peered":"2024-03-26T21:28:40.888796+0000","last_unstale":"2024-03-26T21:53:58.642359+0000","last_undegraded":"2024-03-26T21:53:58.642359+0000","last_fullsized":"2024-03-26T21:53:58.642359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.382983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.382983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:53.425131+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":415,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664900+0000","last_change":"2024-03-26T21:52:50.160918+0000","last_active":"2024-03-26T21:53:59.664900+0000","last_peered":"2024-03-26T21:53:59.664900+0000","last_clean":"2024-03-26T21:53:59.664900+0000","last_became_active":"2024-03-26T21:44:49.395578+0000","last_became_peered":"2024-03-26T21:44:49.395578+0000","last_unstale":"2024-03-26T21:53:59.664900+0000","last_undegraded":"2024-03-26T21:53:59.664900+0000","last_fullsized":"2024-03-26T21:53:59.664900+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-26T21:52:50.160698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.160698+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1064,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769317+0000","last_change":"2024-03-26T21:53:24.786646+0000","last_active":"2024-03-26T21:53:59.769317+0000","last_peered":"2024-03-26T21:53:59.769317+0000","last_clean":"2024-03-26T21:53:59.769317+0000","last_became_active":"2024-03-26T21:41:25.609207+0000","last_became_peered":"2024-03-26T21:41:25.609207+0000","last_unstale":"2024-03-26T21:53:59.769317+0000","last_undegraded":"2024-03-26T21:53:59.769317+0000","last_fullsized":"2024-03-26T21:53:59.769317+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.786545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:24.786545+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.786545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:28.804313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":966,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492863+0000","last_change":"2024-03-26T21:53:40.537631+0000","last_active":"2024-03-26T21:53:59.492863+0000","last_peered":"2024-03-26T21:53:59.492863+0000","last_clean":"2024-03-26T21:53:59.492863+0000","last_became_active":"2024-03-26T21:41:25.621843+0000","last_became_peered":"2024-03-26T21:41:25.621843+0000","last_unstale":"2024-03-26T21:53:59.492863+0000","last_undegraded":"2024-03-26T21:53:59.492863+0000","last_fullsized":"2024-03-26T21:53:59.492863+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:40.537460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:40.537460+0000","last_clean_scrub_stamp":"2024-03-26T21:53:40.537460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.728488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":391,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665404+0000","last_change":"2024-03-26T21:52:41.284391+0000","last_active":"2024-03-26T21:53:59.665404+0000","last_peered":"2024-03-26T21:53:59.665404+0000","last_clean":"2024-03-26T21:53:59.665404+0000","last_became_active":"2024-03-26T21:44:49.419167+0000","last_became_peered":"2024-03-26T21:44:49.419167+0000","last_unstale":"2024-03-26T21:53:59.665404+0000","last_undegraded":"2024-03-26T21:53:59.665404+0000","last_fullsized":"2024-03-26T21:53:59.665404+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:41.284262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:29.844710+0000","last_clean_scrub_stamp":"2024-03-26T21:52:41.284262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:00.239810+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"456'12","reported_seq":411,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:01.953132+0000","last_change":"2024-03-26T21:54:01.953180+0000","last_active":"2024-03-26T21:54:01.953132+0000","last_peered":"2024-03-26T21:54:01.953132+0000","last_clean":"2024-03-26T21:54:01.953132+0000","last_became_active":"2024-03-26T21:44:49.428349+0000","last_became_peered":"2024-03-26T21:44:49.428349+0000","last_unstale":"2024-03-26T21:54:01.953132+0000","last_undegraded":"2024-03-26T21:54:01.953132+0000","last_fullsized":"2024-03-26T21:54:01.953132+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'12","last_scrub_stamp":"2024-03-26T21:54:01.953057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:54:01.953057+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:19.555181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1051,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643349+0000","last_change":"2024-03-26T21:53:40.265060+0000","last_active":"2024-03-26T21:53:58.643349+0000","last_peered":"2024-03-26T21:53:58.643349+0000","last_clean":"2024-03-26T21:53:58.643349+0000","last_became_active":"2024-03-26T21:44:49.439333+0000","last_became_peered":"2024-03-26T21:44:49.439333+0000","last_unstale":"2024-03-26T21:53:58.643349+0000","last_undegraded":"2024-03-26T21:53:58.643349+0000","last_fullsized":"2024-03-26T21:53:58.643349+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:40.264945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:40.264945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:41.943230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":952,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822515+0000","last_change":"2024-03-26T21:53:57.420731+0000","last_active":"2024-03-26T21:53:59.822515+0000","last_peered":"2024-03-26T21:53:59.822515+0000","last_clean":"2024-03-26T21:53:59.822515+0000","last_became_active":"2024-03-26T21:53:02.593001+0000","last_became_peered":"2024-03-26T21:53:02.593001+0000","last_unstale":"2024-03-26T21:53:59.822515+0000","last_undegraded":"2024-03-26T21:53:59.822515+0000","last_fullsized":"2024-03-26T21:53:59.822515+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:57.420579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:26.476531+0000","last_clean_scrub_stamp":"2024-03-26T21:53:57.420579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:25.033198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":986,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.492790+0000","last_change":"2024-03-26T21:53:20.690699+0000","last_active":"2024-03-26T21:53:59.492790+0000","last_peered":"2024-03-26T21:53:59.492790+0000","last_clean":"2024-03-26T21:53:59.492790+0000","last_became_active":"2024-03-26T21:52:08.296169+0000","last_became_peered":"2024-03-26T21:52:08.296169+0000","last_unstale":"2024-03-26T21:53:59.492790+0000","last_undegraded":"2024-03-26T21:53:59.492790+0000","last_fullsized":"2024-03-26T21:53:59.492790+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:20.690575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:20.690575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:30.897935+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":984,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.822975+0000","last_change":"2024-03-26T21:53:18.354751+0000","last_active":"2024-03-26T21:53:59.822975+0000","last_peered":"2024-03-26T21:53:59.822975+0000","last_clean":"2024-03-26T21:53:59.822975+0000","last_became_active":"2024-03-26T21:53:02.594695+0000","last_became_peered":"2024-03-26T21:53:02.594695+0000","last_unstale":"2024-03-26T21:53:59.822975+0000","last_undegraded":"2024-03-26T21:53:59.822975+0000","last_fullsized":"2024-03-26T21:53:59.822975+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-26T21:53:18.354628+0000","last_deep_scrub":"456'32","last_deep_scrub_stamp":"2024-03-26T21:52:04.722406+0000","last_clean_scrub_stamp":"2024-03-26T21:53:18.354628+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:48.280649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":391,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:02.974019+0000","last_change":"2024-03-26T21:54:02.974041+0000","last_active":"2024-03-26T21:54:02.974019+0000","last_peered":"2024-03-26T21:54:02.974019+0000","last_clean":"2024-03-26T21:54:02.974019+0000","last_became_active":"2024-03-26T21:44:49.425920+0000","last_became_peered":"2024-03-26T21:44:49.425920+0000","last_unstale":"2024-03-26T21:54:02.974019+0000","last_undegraded":"2024-03-26T21:54:02.974019+0000","last_fullsized":"2024-03-26T21:54:02.974019+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:54:02.973961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:54:02.973961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:17.244453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643779+0000","last_change":"2024-03-26T21:52:57.406338+0000","last_active":"2024-03-26T21:53:58.643779+0000","last_peered":"2024-03-26T21:53:58.643779+0000","last_clean":"2024-03-26T21:53:58.643779+0000","last_became_active":"2024-03-26T21:52:08.299914+0000","last_became_peered":"2024-03-26T21:52:08.299914+0000","last_unstale":"2024-03-26T21:53:58.643779+0000","last_undegraded":"2024-03-26T21:53:58.643779+0000","last_fullsized":"2024-03-26T21:53:58.643779+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:57.406234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:48:45.073486+0000","last_clean_scrub_stamp":"2024-03-26T21:52:57.406234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:14.651364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1066,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.426402+0000","last_change":"2024-03-26T21:53:55.595005+0000","last_active":"2024-03-26T21:53:59.426402+0000","last_peered":"2024-03-26T21:53:59.426402+0000","last_clean":"2024-03-26T21:53:59.426402+0000","last_became_active":"2024-03-26T21:36:51.712367+0000","last_became_peered":"2024-03-26T21:36:51.712367+0000","last_unstale":"2024-03-26T21:53:59.426402+0000","last_undegraded":"2024-03-26T21:53:59.426402+0000","last_fullsized":"2024-03-26T21:53:59.426402+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:55.594874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:36.551754+0000","last_clean_scrub_stamp":"2024-03-26T21:53:55.594874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:14.351591+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643663+0000","last_change":"2024-03-26T21:53:44.295771+0000","last_active":"2024-03-26T21:53:58.643663+0000","last_peered":"2024-03-26T21:53:58.643663+0000","last_clean":"2024-03-26T21:53:58.643663+0000","last_became_active":"2024-03-26T21:44:49.439907+0000","last_became_peered":"2024-03-26T21:44:49.439907+0000","last_unstale":"2024-03-26T21:53:58.643663+0000","last_undegraded":"2024-03-26T21:53:58.643663+0000","last_fullsized":"2024-03-26T21:53:58.643663+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-26T21:53:44.295661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:44.295661+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:54.251014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'2","reported_seq":1100,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425438+0000","last_change":"2024-03-26T21:53:30.330738+0000","last_active":"2024-03-26T21:53:59.425438+0000","last_peered":"2024-03-26T21:53:59.425438+0000","last_clean":"2024-03-26T21:53:59.425438+0000","last_became_active":"2024-03-26T21:44:49.449941+0000","last_became_peered":"2024-03-26T21:44:49.449941+0000","last_unstale":"2024-03-26T21:53:59.425438+0000","last_undegraded":"2024-03-26T21:53:59.425438+0000","last_fullsized":"2024-03-26T21:53:59.425438+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-26T21:53:30.330651+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-26T21:30:01.058714+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.330651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.444222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"531'16","reported_seq":1232,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.823148+0000","last_change":"2024-03-26T21:53:02.593365+0000","last_active":"2024-03-26T21:53:59.823148+0000","last_peered":"2024-03-26T21:53:59.823148+0000","last_clean":"2024-03-26T21:53:59.823148+0000","last_became_active":"2024-03-26T21:53:02.592915+0000","last_became_peered":"2024-03-26T21:53:02.592915+0000","last_unstale":"2024-03-26T21:53:59.823148+0000","last_undegraded":"2024-03-26T21:53:59.823148+0000","last_fullsized":"2024-03-26T21:53:59.823148+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"468'15","last_scrub_stamp":"2024-03-26T21:52:50.727708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:52:50.727708+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.583946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":990,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.493116+0000","last_change":"2024-03-26T21:53:21.676383+0000","last_active":"2024-03-26T21:53:59.493116+0000","last_peered":"2024-03-26T21:53:59.493116+0000","last_clean":"2024-03-26T21:53:59.493116+0000","last_became_active":"2024-03-26T21:52:08.297191+0000","last_became_peered":"2024-03-26T21:52:08.297191+0000","last_unstale":"2024-03-26T21:53:59.493116+0000","last_undegraded":"2024-03-26T21:53:59.493116+0000","last_fullsized":"2024-03-26T21:53:59.493116+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-26T21:53:21.676270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:21.676270+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.217592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'29","reported_seq":417,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665292+0000","last_change":"2024-03-26T21:52:51.140833+0000","last_active":"2024-03-26T21:53:59.665292+0000","last_peered":"2024-03-26T21:53:59.665292+0000","last_clean":"2024-03-26T21:53:59.665292+0000","last_became_active":"2024-03-26T21:44:49.432922+0000","last_became_peered":"2024-03-26T21:44:49.432922+0000","last_unstale":"2024-03-26T21:53:59.665292+0000","last_undegraded":"2024-03-26T21:53:59.665292+0000","last_fullsized":"2024-03-26T21:53:59.665292+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-26T21:52:51.140738+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-26T21:52:51.140738+0000","last_clean_scrub_stamp":"2024-03-26T21:52:51.140738+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:08.973250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":749,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422279+0000","last_change":"2024-03-26T21:53:55.711499+0000","last_active":"2024-03-26T21:53:59.422279+0000","last_peered":"2024-03-26T21:53:59.422279+0000","last_clean":"2024-03-26T21:53:59.422279+0000","last_became_active":"2024-03-26T21:52:08.303451+0000","last_became_peered":"2024-03-26T21:52:08.303451+0000","last_unstale":"2024-03-26T21:53:59.422279+0000","last_undegraded":"2024-03-26T21:53:59.422279+0000","last_fullsized":"2024-03-26T21:53:59.422279+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:55.711401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:55.711401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:04.060875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":773,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422943+0000","last_change":"2024-03-26T21:53:39.562066+0000","last_active":"2024-03-26T21:53:59.422943+0000","last_peered":"2024-03-26T21:53:59.422943+0000","last_clean":"2024-03-26T21:53:59.422943+0000","last_became_active":"2024-03-26T21:52:08.303646+0000","last_became_peered":"2024-03-26T21:52:08.303646+0000","last_unstale":"2024-03-26T21:53:59.422943+0000","last_undegraded":"2024-03-26T21:53:59.422943+0000","last_fullsized":"2024-03-26T21:53:59.422943+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:39.561927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:39:42.847981+0000","last_clean_scrub_stamp":"2024-03-26T21:53:39.561927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.143555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":384,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665711+0000","last_change":"2024-03-26T21:52:19.244877+0000","last_active":"2024-03-26T21:53:59.665711+0000","last_peered":"2024-03-26T21:53:59.665711+0000","last_clean":"2024-03-26T21:53:59.665711+0000","last_became_active":"2024-03-26T21:44:49.395812+0000","last_became_peered":"2024-03-26T21:44:49.395812+0000","last_unstale":"2024-03-26T21:53:59.665711+0000","last_undegraded":"2024-03-26T21:53:59.665711+0000","last_fullsized":"2024-03-26T21:53:59.665711+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:19.244767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:19.244767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"531'46","reported_seq":114,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.424714+0000","last_change":"2024-03-26T21:53:06.705275+0000","last_active":"2024-03-26T21:53:59.424714+0000","last_peered":"2024-03-26T21:53:59.424714+0000","last_clean":"2024-03-26T21:53:59.424714+0000","last_became_active":"2024-03-26T21:52:08.306770+0000","last_became_peered":"2024-03-26T21:52:08.306770+0000","last_unstale":"2024-03-26T21:53:59.424714+0000","last_undegraded":"2024-03-26T21:53:59.424714+0000","last_fullsized":"2024-03-26T21:53:59.424714+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'46","last_scrub_stamp":"2024-03-26T21:53:06.705083+0000","last_deep_scrub":"531'46","last_deep_scrub_stamp":"2024-03-26T21:53:06.705083+0000","last_clean_scrub_stamp":"2024-03-26T21:53:06.705083+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:17.738937+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643276+0000","last_change":"2024-03-26T21:53:03.375639+0000","last_active":"2024-03-26T21:53:58.643276+0000","last_peered":"2024-03-26T21:53:58.643276+0000","last_clean":"2024-03-26T21:53:58.643276+0000","last_became_active":"2024-03-26T21:41:25.609283+0000","last_became_peered":"2024-03-26T21:41:25.609283+0000","last_unstale":"2024-03-26T21:53:58.643276+0000","last_undegraded":"2024-03-26T21:53:58.643276+0000","last_fullsized":"2024-03-26T21:53:58.643276+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:03.375556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:42:56.212112+0000","last_clean_scrub_stamp":"2024-03-26T21:53:03.375556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:15.145507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1062,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.426512+0000","last_change":"2024-03-26T21:53:41.251121+0000","last_active":"2024-03-26T21:53:59.426512+0000","last_peered":"2024-03-26T21:53:59.426512+0000","last_clean":"2024-03-26T21:53:59.426512+0000","last_became_active":"2024-03-26T21:36:51.710818+0000","last_became_peered":"2024-03-26T21:36:51.710818+0000","last_unstale":"2024-03-26T21:53:59.426512+0000","last_undegraded":"2024-03-26T21:53:59.426512+0000","last_fullsized":"2024-03-26T21:53:59.426512+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:41.251038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:41.251038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:09.822942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":731,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.423162+0000","last_change":"2024-03-26T21:52:49.747504+0000","last_active":"2024-03-26T21:53:59.423162+0000","last_peered":"2024-03-26T21:53:59.423162+0000","last_clean":"2024-03-26T21:53:59.423162+0000","last_became_active":"2024-03-26T21:36:51.685659+0000","last_became_peered":"2024-03-26T21:36:51.685659+0000","last_unstale":"2024-03-26T21:53:59.423162+0000","last_undegraded":"2024-03-26T21:53:59.423162+0000","last_fullsized":"2024-03-26T21:53:59.423162+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:52:49.747390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:52:49.747390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:11.171018+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":773,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422618+0000","last_change":"2024-03-26T21:53:43.598272+0000","last_active":"2024-03-26T21:53:59.422618+0000","last_peered":"2024-03-26T21:53:59.422618+0000","last_clean":"2024-03-26T21:53:59.422618+0000","last_became_active":"2024-03-26T21:44:49.394734+0000","last_became_peered":"2024-03-26T21:44:49.394734+0000","last_unstale":"2024-03-26T21:53:59.422618+0000","last_undegraded":"2024-03-26T21:53:59.422618+0000","last_fullsized":"2024-03-26T21:53:59.422618+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-26T21:53:43.598165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:43.598165+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:47.410698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1076,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:54:01.791116+0000","last_change":"2024-03-26T21:54:01.791155+0000","last_active":"2024-03-26T21:54:01.791116+0000","last_peered":"2024-03-26T21:54:01.791116+0000","last_clean":"2024-03-26T21:54:01.791116+0000","last_became_active":"2024-03-26T21:36:51.677264+0000","last_became_peered":"2024-03-26T21:36:51.677264+0000","last_unstale":"2024-03-26T21:54:01.791116+0000","last_undegraded":"2024-03-26T21:54:01.791116+0000","last_fullsized":"2024-03-26T21:54:01.791116+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:54:01.791046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:47:17.665425+0000","last_clean_scrub_stamp":"2024-03-26T21:54:01.791046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:55:13.434772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642792+0000","last_change":"2024-03-26T21:53:08.390211+0000","last_active":"2024-03-26T21:53:58.642792+0000","last_peered":"2024-03-26T21:53:58.642792+0000","last_clean":"2024-03-26T21:53:58.642792+0000","last_became_active":"2024-03-26T21:51:57.556255+0000","last_became_peered":"2024-03-26T21:51:57.556255+0000","last_unstale":"2024-03-26T21:53:58.642792+0000","last_undegraded":"2024-03-26T21:53:58.642792+0000","last_fullsized":"2024-03-26T21:53:58.642792+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:08.390084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:08.390084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:34.423654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1055,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.768804+0000","last_change":"2024-03-26T21:53:19.868514+0000","last_active":"2024-03-26T21:53:59.768804+0000","last_peered":"2024-03-26T21:53:59.768804+0000","last_clean":"2024-03-26T21:53:59.768804+0000","last_became_active":"2024-03-26T21:44:49.334230+0000","last_became_peered":"2024-03-26T21:44:49.334230+0000","last_unstale":"2024-03-26T21:53:59.768804+0000","last_undegraded":"2024-03-26T21:53:59.768804+0000","last_fullsized":"2024-03-26T21:53:59.768804+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:19.868402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:30:11.468385+0000","last_clean_scrub_stamp":"2024-03-26T21:53:19.868402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:46.602040+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":797,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422585+0000","last_change":"2024-03-26T21:53:00.655476+0000","last_active":"2024-03-26T21:53:59.422585+0000","last_peered":"2024-03-26T21:53:59.422585+0000","last_clean":"2024-03-26T21:53:59.422585+0000","last_became_active":"2024-03-26T21:44:49.395133+0000","last_became_peered":"2024-03-26T21:44:49.395133+0000","last_unstale":"2024-03-26T21:53:59.422585+0000","last_undegraded":"2024-03-26T21:53:59.422585+0000","last_fullsized":"2024-03-26T21:53:59.422585+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-26T21:53:00.655364+0000","last_deep_scrub":"167'13","last_deep_scrub_stamp":"2024-03-26T21:34:30.923562+0000","last_clean_scrub_stamp":"2024-03-26T21:53:00.655364+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:18.067394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1076,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.769256+0000","last_change":"2024-03-26T21:53:35.795881+0000","last_active":"2024-03-26T21:53:59.769256+0000","last_peered":"2024-03-26T21:53:59.769256+0000","last_clean":"2024-03-26T21:53:59.769256+0000","last_became_active":"2024-03-26T21:36:51.700765+0000","last_became_peered":"2024-03-26T21:36:51.700765+0000","last_unstale":"2024-03-26T21:53:59.769256+0000","last_undegraded":"2024-03-26T21:53:59.769256+0000","last_fullsized":"2024-03-26T21:53:59.769256+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:35.795792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:35:48.967897+0000","last_clean_scrub_stamp":"2024-03-26T21:53:35.795792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:49.161500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1056,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.768997+0000","last_change":"2024-03-26T21:53:57.738421+0000","last_active":"2024-03-26T21:53:59.768997+0000","last_peered":"2024-03-26T21:53:59.768997+0000","last_clean":"2024-03-26T21:53:59.768997+0000","last_became_active":"2024-03-26T21:41:25.609825+0000","last_became_peered":"2024-03-26T21:41:25.609825+0000","last_unstale":"2024-03-26T21:53:59.768997+0000","last_undegraded":"2024-03-26T21:53:59.768997+0000","last_fullsized":"2024-03-26T21:53:59.768997+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:57.738315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:50:06.681696+0000","last_clean_scrub_stamp":"2024-03-26T21:53:57.738315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.536418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.642602+0000","last_change":"2024-03-26T21:53:24.430260+0000","last_active":"2024-03-26T21:53:58.642602+0000","last_peered":"2024-03-26T21:53:58.642602+0000","last_clean":"2024-03-26T21:53:58.642602+0000","last_became_active":"2024-03-26T21:41:25.609617+0000","last_became_peered":"2024-03-26T21:41:25.609617+0000","last_unstale":"2024-03-26T21:53:58.642602+0000","last_undegraded":"2024-03-26T21:53:58.642602+0000","last_fullsized":"2024-03-26T21:53:58.642602+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:24.430140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:39.845082+0000","last_clean_scrub_stamp":"2024-03-26T21:53:24.430140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:32.565270+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":105,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.768662+0000","last_change":"2024-03-26T21:53:42.785539+0000","last_active":"2024-03-26T21:53:59.768662+0000","last_peered":"2024-03-26T21:53:59.768662+0000","last_clean":"2024-03-26T21:53:59.768662+0000","last_became_active":"2024-03-26T21:52:17.976213+0000","last_became_peered":"2024-03-26T21:52:17.976213+0000","last_unstale":"2024-03-26T21:53:59.768662+0000","last_undegraded":"2024-03-26T21:53:59.768662+0000","last_fullsized":"2024-03-26T21:53:59.768662+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-26T21:53:42.785401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:42.785401+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:53.180851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":404,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.665367+0000","last_change":"2024-03-26T21:53:42.153289+0000","last_active":"2024-03-26T21:53:59.665367+0000","last_peered":"2024-03-26T21:53:59.665367+0000","last_clean":"2024-03-26T21:53:59.665367+0000","last_became_active":"2024-03-26T21:52:08.303017+0000","last_became_peered":"2024-03-26T21:52:08.303017+0000","last_unstale":"2024-03-26T21:53:59.665367+0000","last_undegraded":"2024-03-26T21:53:59.665367+0000","last_fullsized":"2024-03-26T21:53:59.665367+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:42.153207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:33.816134+0000","last_clean_scrub_stamp":"2024-03-26T21:53:42.153207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:58.008511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":738,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.421873+0000","last_change":"2024-03-26T21:53:22.680583+0000","last_active":"2024-03-26T21:53:59.421873+0000","last_peered":"2024-03-26T21:53:59.421873+0000","last_clean":"2024-03-26T21:53:59.421873+0000","last_became_active":"2024-03-26T21:36:51.695476+0000","last_became_peered":"2024-03-26T21:36:51.695476+0000","last_unstale":"2024-03-26T21:53:59.421873+0000","last_undegraded":"2024-03-26T21:53:59.421873+0000","last_fullsized":"2024-03-26T21:53:59.421873+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:22.680471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:22.680471+0000","last_clean_scrub_stamp":"2024-03-26T21:53:22.680471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:50.861277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":391,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.664755+0000","last_change":"2024-03-26T21:53:30.378126+0000","last_active":"2024-03-26T21:53:59.664755+0000","last_peered":"2024-03-26T21:53:59.664755+0000","last_clean":"2024-03-26T21:53:59.664755+0000","last_became_active":"2024-03-26T21:44:49.396478+0000","last_became_peered":"2024-03-26T21:44:49.396478+0000","last_unstale":"2024-03-26T21:53:59.664755+0000","last_undegraded":"2024-03-26T21:53:59.664755+0000","last_fullsized":"2024-03-26T21:53:59.664755+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:30.378020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:34:24.368161+0000","last_clean_scrub_stamp":"2024-03-26T21:53:30.378020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:34.529284+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":782,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422008+0000","last_change":"2024-03-26T21:53:33.622669+0000","last_active":"2024-03-26T21:53:59.422008+0000","last_peered":"2024-03-26T21:53:59.422008+0000","last_clean":"2024-03-26T21:53:59.422008+0000","last_became_active":"2024-03-26T21:43:16.796048+0000","last_became_peered":"2024-03-26T21:43:16.796048+0000","last_unstale":"2024-03-26T21:53:59.422008+0000","last_undegraded":"2024-03-26T21:53:59.422008+0000","last_fullsized":"2024-03-26T21:53:59.422008+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-26T21:53:33.622527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:35.824090+0000","last_clean_scrub_stamp":"2024-03-26T21:53:33.622527+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:40.365986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1053,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-26T21:53:58.643136+0000","last_change":"2024-03-26T21:53:28.377735+0000","last_active":"2024-03-26T21:53:58.643136+0000","last_peered":"2024-03-26T21:53:58.643136+0000","last_clean":"2024-03-26T21:53:58.643136+0000","last_became_active":"2024-03-26T21:52:08.299872+0000","last_became_peered":"2024-03-26T21:52:08.299872+0000","last_unstale":"2024-03-26T21:53:58.643136+0000","last_undegraded":"2024-03-26T21:53:58.643136+0000","last_fullsized":"2024-03-26T21:53:58.643136+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:28.377651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:52:27.414633+0000","last_clean_scrub_stamp":"2024-03-26T21:53:28.377651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:54.328768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1050,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.425254+0000","last_change":"2024-03-26T21:53:23.273559+0000","last_active":"2024-03-26T21:53:59.425254+0000","last_peered":"2024-03-26T21:53:59.425254+0000","last_clean":"2024-03-26T21:53:59.425254+0000","last_became_active":"2024-03-26T21:41:25.609733+0000","last_became_peered":"2024-03-26T21:41:25.609733+0000","last_unstale":"2024-03-26T21:53:59.425254+0000","last_undegraded":"2024-03-26T21:53:59.425254+0000","last_fullsized":"2024-03-26T21:53:59.425254+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:23.273480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:28:37.828962+0000","last_clean_scrub_stamp":"2024-03-26T21:53:23.273480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:42.468181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":737,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-26T21:53:59.422412+0000","last_change":"2024-03-26T21:53:01.728655+0000","last_active":"2024-03-26T21:53:59.422412+0000","last_peered":"2024-03-26T21:53:59.422412+0000","last_clean":"2024-03-26T21:53:59.422412+0000","last_became_active":"2024-03-26T21:36:51.700576+0000","last_became_peered":"2024-03-26T21:36:51.700576+0000","last_unstale":"2024-03-26T21:53:59.422412+0000","last_undegraded":"2024-03-26T21:53:59.422412+0000","last_fullsized":"2024-03-26T21:53:59.422412+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T21:53:01.728552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T21:53:01.728552+0000","last_clean_scrub_stamp":"2024-03-26T21:53:01.728552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T21:54:11.299636+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1407,"num_read_kb":1402,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":8038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},"log_size":1098,"ondisk_log_size":1098,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":276,"seq":1185410973850,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596300,"kb_used_data":1340,"kb_used_omap":6,"kb_used_meta":594937,"kb_avail":93148852,"statfs":{"total":95995035648,"available":95384424448,"internally_reserved":0,"allocated":1372160,"data_stored":967261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6872,"internal_metadata":609215784},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365117,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732200,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93012952,"statfs":{"total":95995035648,"available":95245262848,"internally_reserved":0,"allocated":1339392,"data_stored":964849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575191,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974596,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92770556,"statfs":{"total":95995035648,"available":94997049344,"internally_reserved":0,"allocated":782336,"data_stored":375904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738715,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939064,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806088,"statfs":{"total":95995035648,"available":95033434112,"internally_reserved":0,"allocated":1359872,"data_stored":953451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411665,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735784,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009368,"statfs":{"total":95995035648,"available":95241592832,"internally_reserved":0,"allocated":745472,"data_stored":374706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150027,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988484,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92756668,"statfs":{"total":95995035648,"available":94982828032,"internally_reserved":0,"allocated":782336,"data_stored":375275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1011415495},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":133,"seq":571230650623,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817288,"kb_used_data":764,"kb_used_omap":4,"kb_used_meta":816507,"kb_avail":92927864,"statfs":{"total":95995035648,"available":95158132736,"internally_reserved":0,"allocated":782336,"data_stored":376433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":836103843},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659456,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735808,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":93009344,"statfs":{"total":95995035648,"available":95241568256,"internally_reserved":0,"allocated":770048,"data_stored":375275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":752679365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2429,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T21:54:06.679 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T21:54:06.679 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55 shell --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-26T21:54:06.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.817 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[40587]: 5.15 scrub starts 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[40587]: 5.15 scrub ok 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[29406]: 5.15 scrub starts 2024-03-26T21:54:06.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:06 smithi055 ceph-mon[29406]: 5.15 scrub ok 2024-03-26T21:54:06.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:06 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:06 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:06.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:06 smithi067 ceph-mon[33346]: 5.15 scrub starts 2024-03-26T21:54:06.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:06 smithi067 ceph-mon[33346]: 5.15 scrub ok 2024-03-26T21:54:07.609 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T21:54:07.732 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: 6.3 scrub starts 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: 6.3 scrub ok 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: 6.7 scrub starts 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[29406]: 6.7 scrub ok 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: 6.3 scrub starts 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: 6.3 scrub ok 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: 6.7 scrub starts 2024-03-26T21:54:07.733 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:07 smithi055 ceph-mon[40587]: 6.7 scrub ok 2024-03-26T21:54:07.960 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T21:54:07.960 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T21:54:07.961 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: 6.3 scrub starts 2024-03-26T21:54:07.961 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: 6.3 scrub ok 2024-03-26T21:54:07.961 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: 6.7 scrub starts 2024-03-26T21:54:07.961 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-mon[33346]: 6.7 scrub ok 2024-03-26T21:54:08.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:54:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug there is no tcmu-runner data available 2024-03-26T21:54:08.817 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: 3.13 scrub starts 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: 3.13 scrub ok 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: 5.13 deep-scrub starts 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: 5.13 deep-scrub ok 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.818 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: 3.13 scrub starts 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: 3.13 scrub ok 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: 5.13 deep-scrub starts 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: 5.13 deep-scrub ok 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:08.819 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: 3.13 scrub starts 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: 3.13 scrub ok 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: 5.13 deep-scrub starts 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: 5.13 deep-scrub ok 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:08.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:08 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:09.205 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:08 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: ::ffff:172.21.15.67 - - [26/Mar/2024:21:54:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 3.a deep-scrub starts 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 3.a deep-scrub ok 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 5.f scrub starts 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 5.f scrub ok 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 6.c scrub starts 2024-03-26T21:54:09.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:09 smithi067 ceph-mon[33346]: 6.c scrub ok 2024-03-26T21:54:10.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 3.a deep-scrub starts 2024-03-26T21:54:10.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 3.a deep-scrub ok 2024-03-26T21:54:10.067 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 5.f scrub starts 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 5.f scrub ok 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 6.c scrub starts 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[29406]: 6.c scrub ok 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 3.a deep-scrub starts 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 3.a deep-scrub ok 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 5.f scrub starts 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 5.f scrub ok 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 6.c scrub starts 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-mon[40587]: 6.c scrub ok 2024-03-26T21:54:10.068 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE Bus STOPPING 2024-03-26T21:54:10.069 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:54:10.069 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE Bus STOPPED 2024-03-26T21:54:10.069 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE Bus STARTING 2024-03-26T21:54:10.069 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE Serving on http://:::9283 2024-03-26T21:54:10.069 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:09 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:09] ENGINE Bus STARTED 2024-03-26T21:54:10.408 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T21:54:10.417 INFO:tasks.cephadm:Teardown begin 2024-03-26T21:54:10.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:54:10.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:54:10.533 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T21:54:10.533 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:54:10.584 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T21:54:10.607 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T21:54:10.607 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T21:54:10.607 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a 2024-03-26T21:54:10.633 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T21:54:10.633 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T21:54:10.633 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:10.633 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:10.633 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[40587]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:10.634 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-mon[29406]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-26T21:54:10.901 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 systemd[1]: Stopping Ceph mon.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:10.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T21:54:10.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='client.? 172.21.15.55:0/1888530693' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T21:54:10.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T21:54:10.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T21:54:10.991 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='mgr.24470 172.21.15.55:0/2157782315' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T21:54:10.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: from='mgr.24470 ' entity='mgr.y' 2024-03-26T21:54:10.992 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:10 smithi067 ceph-mon[33346]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-26T21:54:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:54:10.905+0000 7fb204c51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:11.317 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:10 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a[29402]: 2024-03-26T21:54:10.905+0000 7fb204c51700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T21:54:11.318 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:11 smithi055 podman[217709]: 2024-03-26 21:54:11.110306837 +0000 UTC m=+0.298842156 container died a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=) 2024-03-26T21:54:11.735 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:11 smithi055 podman[217709]: 2024-03-26 21:54:11.464580531 +0000 UTC m=+0.653115846 container cleanup a292aaadcb66b2a0d3ee72180756720ae47417e35ddc9b717a18d8020784f2e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T21:54:11.735 INFO:journalctl@ceph.mon.a.smithi055.stdout:Mar 26 21:54:11 smithi055 bash[217709]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-a 2024-03-26T21:54:11.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T21:54:11.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:11.963 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T21:54:11.963 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T21:54:11.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.c 2024-03-26T21:54:12.274 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 systemd[1]: Stopping Ceph mon.c for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:12.567 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c[40583]: 2024-03-26T21:54:12.279+0000 7f98ee8d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c[40583]: 2024-03-26T21:54:12.279+0000 7f98ee8d7700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 podman[217830]: 2024-03-26 21:54:12.310794025 +0000 UTC m=+0.148948447 container died dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE Bus STOPPING 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE Bus STOPPED 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE Bus STARTING 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE Serving on http://:::9283 2024-03-26T21:54:12.568 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:12 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: [26/Mar/2024:21:54:12] ENGINE Bus STARTED 2024-03-26T21:54:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 podman[217830]: 2024-03-26 21:54:12.771794956 +0000 UTC m=+0.609949420 container cleanup dfb30b8e027b4aff94a9fbfd921fd680076abcd3641dda05a85c0935e7644686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:54:13.059 INFO:journalctl@ceph.mon.c.smithi055.stdout:Mar 26 21:54:12 smithi055 bash[217830]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-c 2024-03-26T21:54:13.193 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T21:54:13.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:13.230 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T21:54:13.230 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T21:54:13.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.b 2024-03-26T21:54:13.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:13 smithi067 systemd[1]: Stopping Ceph mon.b for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:13.850 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-b[33342]: 2024-03-26T21:54:13.578+0000 7f1ea8a87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:13.850 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-b[33342]: 2024-03-26T21:54:13.578+0000 7f1ea8a87700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T21:54:13.850 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:13 smithi067 podman[165532]: 2024-03-26 21:54:13.802829787 +0000 UTC m=+0.330167943 container died 84744af93b1ff22e9abb3f2ccbc05e17970a7f02e9d172140304ca1ed22e8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:54:14.480 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:14 smithi067 podman[165532]: 2024-03-26 21:54:14.167238656 +0000 UTC m=+0.694576818 container cleanup 84744af93b1ff22e9abb3f2ccbc05e17970a7f02e9d172140304ca1ed22e8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-b, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True) 2024-03-26T21:54:14.480 INFO:journalctl@ceph.mon.b.smithi067.stdout:Mar 26 21:54:14 smithi067 bash[165532]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mon-b 2024-03-26T21:54:14.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T21:54:14.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:14.647 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T21:54:14.647 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T21:54:14.648 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y 2024-03-26T21:54:14.946 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:14 smithi055 systemd[1]: Stopping Ceph mgr.y for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:15.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:14 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:54:14.950+0000 7ffa2b581700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:15.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:14 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y[29641]: 2024-03-26T21:54:14.950+0000 7ffa2b581700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T21:54:15.318 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:14 smithi055 podman[217961]: 2024-03-26 21:54:14.982032415 +0000 UTC m=+0.145504924 container died 39119b4228e67f72d444a06ab92bfe5b29557801637957c5c963d82235572891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T21:54:15.817 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:15 smithi055 podman[217961]: 2024-03-26 21:54:15.543399533 +0000 UTC m=+0.706872041 container cleanup 39119b4228e67f72d444a06ab92bfe5b29557801637957c5c963d82235572891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42fc52b86b14d4f88c40d75eb7102e0ab3918f55, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240326, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:54:15.818 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Mar 26 21:54:15 smithi055 bash[217961]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-y 2024-03-26T21:54:15.936 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T21:54:15.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:15.972 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T21:54:15.972 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T21:54:15.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.x 2024-03-26T21:54:16.306 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 systemd[1]: Stopping Ceph mgr.x for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:16.608 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 podman[165667]: 2024-03-26 21:54:16.311458326 +0000 UTC m=+0.140592602 container died da2aa66bf7701b690953e5c9a7fbcaafe6b10835ecbecf9755d7cbdf2d826a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T21:54:16.991 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 podman[165667]: 2024-03-26 21:54:16.613914027 +0000 UTC m=+0.443048302 container cleanup da2aa66bf7701b690953e5c9a7fbcaafe6b10835ecbecf9755d7cbdf2d826a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T21:54:16.991 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 bash[165667]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x 2024-03-26T21:54:16.991 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 podman[165681]: 2024-03-26 21:54:16.798403215 +0000 UTC m=+0.490387788 container remove da2aa66bf7701b690953e5c9a7fbcaafe6b10835ecbecf9755d7cbdf2d826a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-26T21:54:16.991 INFO:journalctl@ceph.mgr.x.smithi067.stdout:Mar 26 21:54:16 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T21:54:17.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T21:54:17.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:17.055 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T21:54:17.056 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T21:54:17.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.0 2024-03-26T21:54:17.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:17 smithi055 systemd[1]: Stopping Ceph osd.0 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:17.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:17 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0[44589]: 2024-03-26T21:54:17.372+0000 7f2986cd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:17.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:17 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0[44589]: 2024-03-26T21:54:17.372+0000 7f2986cd4700 -1 osd.0 541 *** Got signal Terminated *** 2024-03-26T21:54:17.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:17 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0[44589]: 2024-03-26T21:54:17.372+0000 7f2986cd4700 -1 osd.0 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:23.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:22 smithi055 podman[218082]: 2024-03-26 21:54:22.895991636 +0000 UTC m=+5.638967556 container died 1323920da8ce38afc44afeec3a665f70964ef87f3a21ff2b3b38cf39b08ab4c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T21:54:23.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:23 smithi055 podman[218082]: 2024-03-26 21:54:23.33155559 +0000 UTC m=+6.074531550 container cleanup 1323920da8ce38afc44afeec3a665f70964ef87f3a21ff2b3b38cf39b08ab4c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T21:54:23.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:23 smithi055 bash[218082]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0 2024-03-26T21:54:23.817 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:23 smithi055 podman[218098]: 2024-03-26 21:54:23.549811245 +0000 UTC m=+0.650107754 container remove 1323920da8ce38afc44afeec3a665f70964ef87f3a21ff2b3b38cf39b08ab4c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2) 2024-03-26T21:54:24.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:23 smithi055 podman[218195]: 2024-03-26 21:54:23.898315902 +0000 UTC m=+0.017669995 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:54:24.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:24 smithi055 podman[218195]: 2024-03-26 21:54:24.042468009 +0000 UTC m=+0.161822087 container create 755776c0eff8d2c4778b5b4fa459a6a148106328bb8f40b407e9636f6604f828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T21:54:24.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:24 smithi055 podman[218195]: 2024-03-26 21:54:24.261732949 +0000 UTC m=+0.381087027 container init 755776c0eff8d2c4778b5b4fa459a6a148106328bb8f40b407e9636f6604f828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:54:24.567 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:24 smithi055 podman[218195]: 2024-03-26 21:54:24.265047173 +0000 UTC m=+0.384401252 container start 755776c0eff8d2c4778b5b4fa459a6a148106328bb8f40b407e9636f6604f828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-26T21:54:24.568 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:24 smithi055 podman[218195]: 2024-03-26 21:54:24.386783818 +0000 UTC m=+0.506137907 container attach 755776c0eff8d2c4778b5b4fa459a6a148106328bb8f40b407e9636f6604f828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-03-26T21:54:25.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Mar 26 21:54:24 smithi055 podman[218195]: 2024-03-26 21:54:24.818297454 +0000 UTC m=+0.937651539 container died 755776c0eff8d2c4778b5b4fa459a6a148106328bb8f40b407e9636f6604f828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:54:25.458 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T21:54:25.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:25.511 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T21:54:25.511 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T21:54:25.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.1 2024-03-26T21:54:25.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:25 smithi055 systemd[1]: Stopping Ceph osd.1 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:26.067 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1[50597]: 2024-03-26T21:54:25.817+0000 7f3c16a99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:26.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1[50597]: 2024-03-26T21:54:25.817+0000 7f3c16a99700 -1 osd.1 541 *** Got signal Terminated *** 2024-03-26T21:54:26.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:25 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1[50597]: 2024-03-26T21:54:25.817+0000 7f3c16a99700 -1 osd.1 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:31.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:31 smithi055 podman[218310]: 2024-03-26 21:54:31.416812096 +0000 UTC m=+5.714855362 container died 2fdc7c9165281de28838a0970472c6902b99147308d78554285446fb70c85c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T21:54:32.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:31 smithi055 podman[218310]: 2024-03-26 21:54:31.902113994 +0000 UTC m=+6.200157317 container cleanup 2fdc7c9165281de28838a0970472c6902b99147308d78554285446fb70c85c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:54:32.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:31 smithi055 bash[218310]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1 2024-03-26T21:54:32.526 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218331]: 2024-03-26 21:54:32.161831998 +0000 UTC m=+0.742528480 container remove 2fdc7c9165281de28838a0970472c6902b99147308d78554285446fb70c85c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T21:54:32.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218423]: 2024-03-26 21:54:32.435419527 +0000 UTC m=+0.017989811 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:54:32.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218423]: 2024-03-26 21:54:32.562767341 +0000 UTC m=+0.145337598 container create c6400eece4fc78a05e54a30a2e7f2a0812bbd09f56b6f1433206bcb8f21e0763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:54:33.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218423]: 2024-03-26 21:54:32.798371341 +0000 UTC m=+0.380941601 container init c6400eece4fc78a05e54a30a2e7f2a0812bbd09f56b6f1433206bcb8f21e0763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T21:54:33.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218423]: 2024-03-26 21:54:32.801831235 +0000 UTC m=+0.384401494 container start c6400eece4fc78a05e54a30a2e7f2a0812bbd09f56b6f1433206bcb8f21e0763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T21:54:33.068 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:32 smithi055 podman[218423]: 2024-03-26 21:54:32.906889867 +0000 UTC m=+0.489460136 container attach c6400eece4fc78a05e54a30a2e7f2a0812bbd09f56b6f1433206bcb8f21e0763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-26T21:54:33.567 INFO:journalctl@ceph.osd.1.smithi055.stdout:Mar 26 21:54:33 smithi055 podman[218423]: 2024-03-26 21:54:33.099754034 +0000 UTC m=+0.682324302 container died c6400eece4fc78a05e54a30a2e7f2a0812bbd09f56b6f1433206bcb8f21e0763 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-26T21:54:33.741 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T21:54:33.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:33.781 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T21:54:33.781 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T21:54:33.781 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.2 2024-03-26T21:54:34.069 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:33 smithi055 systemd[1]: Stopping Ceph osd.2 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:34.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:34 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2[56336]: 2024-03-26T21:54:34.146+0000 7f8c46bae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:34.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:34 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2[56336]: 2024-03-26T21:54:34.146+0000 7f8c46bae700 -1 osd.2 541 *** Got signal Terminated *** 2024-03-26T21:54:34.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:34 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2[56336]: 2024-03-26T21:54:34.146+0000 7f8c46bae700 -1 osd.2 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:40.010 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:39 smithi055 podman[218529]: 2024-03-26 21:54:39.716347731 +0000 UTC m=+5.710045963 container died f5a51a2dc225b4601a42c440f5bdba2ad96e15cc5ddaa0bdfe1c8536c059a576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:54:40.317 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:40 smithi055 podman[218529]: 2024-03-26 21:54:40.205956233 +0000 UTC m=+6.199668366 container cleanup f5a51a2dc225b4601a42c440f5bdba2ad96e15cc5ddaa0bdfe1c8536c059a576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T21:54:40.318 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:40 smithi055 bash[218529]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2 2024-03-26T21:54:40.815 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:40 smithi055 podman[218548]: 2024-03-26 21:54:40.482589437 +0000 UTC m=+0.749888419 container remove f5a51a2dc225b4601a42c440f5bdba2ad96e15cc5ddaa0bdfe1c8536c059a576 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:54:41.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:40 smithi055 podman[218641]: 2024-03-26 21:54:40.723986798 +0000 UTC m=+0.017612050 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:54:41.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:40 smithi055 podman[218641]: 2024-03-26 21:54:40.833702827 +0000 UTC m=+0.127328068 container create ef42a3bacbb0f97dc3fd7454c085c05234abffafe40ce90c5de3b006bdb7fd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T21:54:41.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:41 smithi055 podman[218641]: 2024-03-26 21:54:41.027666346 +0000 UTC m=+0.321291587 container init ef42a3bacbb0f97dc3fd7454c085c05234abffafe40ce90c5de3b006bdb7fd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True) 2024-03-26T21:54:41.068 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:41 smithi055 podman[218641]: 2024-03-26 21:54:41.03126396 +0000 UTC m=+0.324889202 container start ef42a3bacbb0f97dc3fd7454c085c05234abffafe40ce90c5de3b006bdb7fd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, maintainer=Guillaume Abrioux ) 2024-03-26T21:54:41.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:41 smithi055 podman[218641]: 2024-03-26 21:54:41.119486186 +0000 UTC m=+0.413111426 container attach ef42a3bacbb0f97dc3fd7454c085c05234abffafe40ce90c5de3b006bdb7fd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T21:54:41.568 INFO:journalctl@ceph.osd.2.smithi055.stdout:Mar 26 21:54:41 smithi055 podman[218641]: 2024-03-26 21:54:41.329572268 +0000 UTC m=+0.623197516 container died ef42a3bacbb0f97dc3fd7454c085c05234abffafe40ce90c5de3b006bdb7fd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T21:54:42.015 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T21:54:42.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:42.059 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T21:54:42.059 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T21:54:42.060 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.3 2024-03-26T21:54:42.318 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:42 smithi055 systemd[1]: Stopping Ceph osd.3 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:42.818 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:42 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:54:42.368+0000 7fbbdaa8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:42.818 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:42 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:54:42.368+0000 7fbbdaa8d700 -1 osd.3 541 *** Got signal Terminated *** 2024-03-26T21:54:42.818 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:42 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:54:42.368+0000 7fbbdaa8d700 -1 osd.3 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:43.614 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:43 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:43.251+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:43.987 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:43 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:43.618+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:44.241 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:44.214+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:44.241 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:43 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:43.991+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:44.544 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:54:44.298Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.55:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.55:8765: connect: connection refused" 2024-03-26T21:54:44.544 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:54:44.300Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.55:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.55:8765: connect: connection refused" 2024-03-26T21:54:44.544 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:54:44.301Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.55:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.55:8765: connect: connection refused" 2024-03-26T21:54:44.544 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:54:44.302Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.55:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.55:8765: connect: connection refused" 2024-03-26T21:54:44.544 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:54:44.302Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.55:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.55:8765: connect: connection refused" 2024-03-26T21:54:44.838 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:44.569+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:45.225 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:44.984+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:45.225 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:44 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:44.842+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:45.491 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:45 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:45.229+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:45.826 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:45 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:45.566+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:46.241 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:45 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:45.830+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:46.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:45 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:45.978+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:46.568 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:46.257+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:46.818 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:46 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3[62126]: 2024-03-26T21:54:46.394+0000 7fbbd3079700 -1 osd.3 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.243005+0000 front 2024-03-26T21:54:20.242985+0000 (oldest deadline 2024-03-26T21:54:45.443750+0000) 2024-03-26T21:54:46.967 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:46.806+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:46.968 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:46.573+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:47.241 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:46 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:46.971+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:47.543 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:47.259+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:47.834 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:47.548+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:48.241 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:47.839+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:48.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:47 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:47.966+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:48.318 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:48 smithi055 podman[218746]: 2024-03-26 21:54:48.028459581 +0000 UTC m=+5.771751070 container died 12a166f4779979e335087712f9aa717b6f4f7dcbe4d9042bfa0eb22c0501e3fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-26T21:54:48.556 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:48 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:48.299+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:48.788 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:48 smithi055 podman[218746]: 2024-03-26 21:54:48.509243873 +0000 UTC m=+6.252535404 container cleanup 12a166f4779979e335087712f9aa717b6f4f7dcbe4d9042bfa0eb22c0501e3fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T21:54:48.788 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:48 smithi055 bash[218746]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3 2024-03-26T21:54:48.933 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:48 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:48.796+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:48.934 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:48 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:48.560+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:49.068 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:48 smithi055 podman[218763]: 2024-03-26 21:54:48.793925013 +0000 UTC m=+0.757100384 container remove 12a166f4779979e335087712f9aa717b6f4f7dcbe4d9042bfa0eb22c0501e3fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T21:54:49.241 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:48 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:48.937+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:49.375 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.011135267 +0000 UTC m=+0.018295114 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:54:49.375 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.119669831 +0000 UTC m=+0.126829644 container create bb7be91a0081271d462748e70de66abf136c8045ff8fead4195b302a08deb9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:54:49.375 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.314505978 +0000 UTC m=+0.321665790 container init bb7be91a0081271d462748e70de66abf136c8045ff8fead4195b302a08deb9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T21:54:49.375 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.317747146 +0000 UTC m=+0.324906959 container start bb7be91a0081271d462748e70de66abf136c8045ff8fead4195b302a08deb9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-26T21:54:49.375 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.380217393 +0000 UTC m=+0.387377214 container attach bb7be91a0081271d462748e70de66abf136c8045ff8fead4195b302a08deb9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-26T21:54:49.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:49 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:49.566+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:49.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:49 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:49.318+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:49.818 INFO:journalctl@ceph.osd.3.smithi055.stdout:Mar 26 21:54:49 smithi055 podman[218853]: 2024-03-26 21:54:49.62291389 +0000 UTC m=+0.630073715 container died bb7be91a0081271d462748e70de66abf136c8045ff8fead4195b302a08deb9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T21:54:50.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T21:54:50.241 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:49 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:49.794+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:50.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:49 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:49.895+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:50.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:50.286 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T21:54:50.286 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T21:54:50.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4 2024-03-26T21:54:50.587 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:50.559+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:50.587 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:50.357+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:50.588 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:50 smithi067 systemd[1]: Stopping Ceph osd.4 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:50.932 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:50.814+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:50.932 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:50.591+0000 7f87166bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:50.932 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:50.591+0000 7f87166bf700 -1 osd.4 541 *** Got signal Terminated *** 2024-03-26T21:54:50.932 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:50.591+0000 7f87166bf700 -1 osd.4 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:51.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:50 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:50.936+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:51.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:51 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:51.366+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:51.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:51 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:51.549+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:52.241 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:51 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:51.835+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:52.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:51 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:51.920+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:52.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:52 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:52.516+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:52.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:52 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:52.404+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:53.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:52 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:52.850+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:53.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:52 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:52.959+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:53.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:53.526+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:53.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:53.369+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:54.241 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:53.892+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:54.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:53 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:53.918+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:54.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:54.531+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:54.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:54.362+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:54.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:54.362+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:54:55.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:54.936+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:55.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:54.936+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:54:55.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:54.912+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:21.825433+0000 front 2024-03-26T21:54:21.825406+0000 (oldest deadline 2024-03-26T21:54:43.525281+0000) 2024-03-26T21:54:55.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:54 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4[77168]: 2024-03-26T21:54:54.912+0000 7f8712cb3700 -1 osd.4 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.727079+0000 front 2024-03-26T21:54:28.727102+0000 (oldest deadline 2024-03-26T21:54:54.026785+0000) 2024-03-26T21:54:55.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:55 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:55.520+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:55.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:55 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:55.371+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:55.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:55 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:55.371+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:54:56.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:55 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:55.954+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:56.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:55 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:55.954+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:54:56.242 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:56 smithi067 podman[165806]: 2024-03-26 21:54:56.094105839 +0000 UTC m=+5.601368844 container died 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:56.475+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:56.475+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:56.374+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:56.374+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:56 smithi067 podman[165806]: 2024-03-26 21:54:56.483344857 +0000 UTC m=+5.990607859 container cleanup 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:54:56.742 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:56 smithi067 bash[165806]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4 2024-03-26T21:54:57.103 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:56.936+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:57.104 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:56.937+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:54:57.104 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:56 smithi067 podman[165825]: 2024-03-26 21:54:56.801373489 +0000 UTC m=+0.664872849 container remove 692831a2cb82eb402b9a6a8edc4aeb531f003fad9b70496f0ea2bbbf36e18800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T21:54:57.383 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.012256393 +0000 UTC m=+0.018082613 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:54:57.383 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.143938888 +0000 UTC m=+0.149765084 container create 9254427bd4b9ee86f8d24030b9c1c4daf5d9db1ff6eadcc7217164f763449a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T21:54:57.383 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.334426321 +0000 UTC m=+0.340252519 container init 9254427bd4b9ee86f8d24030b9c1c4daf5d9db1ff6eadcc7217164f763449a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:54:57.383 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.337447778 +0000 UTC m=+0.343273976 container start 9254427bd4b9ee86f8d24030b9c1c4daf5d9db1ff6eadcc7217164f763449a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T21:54:57.741 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.434554121 +0000 UTC m=+0.440380327 container attach 9254427bd4b9ee86f8d24030b9c1c4daf5d9db1ff6eadcc7217164f763449a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T21:54:57.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:57.474+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:57.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:57.474+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:54:57.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:57.387+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:57.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:57.387+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:54:58.082 INFO:journalctl@ceph.osd.4.smithi067.stdout:Mar 26 21:54:57 smithi067 podman[165917]: 2024-03-26 21:54:57.795212107 +0000 UTC m=+0.801038298 container died 9254427bd4b9ee86f8d24030b9c1c4daf5d9db1ff6eadcc7217164f763449a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T21:54:58.083 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:57.922+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:58.083 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:57 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:57.922+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:54:58.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T21:54:58.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:54:58.420 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T21:54:58.420 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T21:54:58.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.5 2024-03-26T21:54:58.659 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 systemd[1]: Stopping Ceph osd.5 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:54:58.659 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:58.448+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:58.660 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:58.448+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:54:58.660 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:58.351+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:58.660 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:58.351+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:54:58.991 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:58.756+0000 7fcfa4457700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:54:58.992 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:58.756+0000 7fcfa4457700 -1 osd.5 541 *** Got signal Terminated *** 2024-03-26T21:54:58.992 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:58.756+0000 7fcfa4457700 -1 osd.5 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:54:58.992 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:58.891+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:54:58.992 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:58 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:58.891+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:54:59.629 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:59.436+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:54:59.630 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:54:59.436+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:54:59.630 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:59.361+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:54:59.630 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:54:59.361+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:00.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:59.932+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:55:00.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:54:59 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:54:59.932+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:55:00.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:00.406+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:00.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:00.406+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:00.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:00.406+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:00.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:00.337+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:00.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:00.337+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:01.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:00.930+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:55:01.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:00 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:00.930+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:55:01.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:01.374+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:01.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:01.374+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:01.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:01.374+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:01.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:01.316+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:01.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:01.316+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:01.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:01.316+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:02.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:01.919+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:55:02.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:01 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:01.919+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:02.344+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:02.344+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:02.345+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:02.326+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:02.326+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:02.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:02.326+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:03.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:02.894+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:20.291746+0000 front 2024-03-26T21:54:20.291630+0000 (oldest deadline 2024-03-26T21:54:44.391457+0000) 2024-03-26T21:55:03.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:02.894+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.893806+0000 front 2024-03-26T21:54:28.893732+0000 (oldest deadline 2024-03-26T21:54:54.793596+0000) 2024-03-26T21:55:03.242 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:02 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5[43872]: 2024-03-26T21:55:02.894+0000 7fcfa0a4b700 -1 osd.5 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:37.694642+0000 front 2024-03-26T21:54:37.694655+0000 (oldest deadline 2024-03-26T21:55:02.394574+0000) 2024-03-26T21:55:03.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:03.394+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:03.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:03.394+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:03.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:03.394+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:03.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:03.323+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:03.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:03.323+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:03.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:03 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:03.323+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:04 smithi067 podman[166024]: 2024-03-26 21:55:04.18866873 +0000 UTC m=+5.520736235 container died c18170f720b0d69865b99163c39ec165d640ba8f12de7001946caa5a297b7838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:04.359+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:04.359+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:04.359+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:04.312+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:04.480 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:04.312+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:04.481 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:04 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:04.312+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:04.742 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:04 smithi067 podman[166024]: 2024-03-26 21:55:04.58280203 +0000 UTC m=+5.914869535 container cleanup c18170f720b0d69865b99163c39ec165d640ba8f12de7001946caa5a297b7838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:55:04.742 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:04 smithi067 bash[166024]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5 2024-03-26T21:55:05.205 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:04 smithi067 podman[166052]: 2024-03-26 21:55:04.8762187 +0000 UTC m=+0.685569541 container remove c18170f720b0d69865b99163c39ec165d640ba8f12de7001946caa5a297b7838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T21:55:05.491 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:05.407+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:05.407+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:05.407+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:05.302+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:05.302+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:05 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:05.302+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:05.492 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.210411468 +0000 UTC m=+0.113631116 container create 162c104546306cf7a97638f1fc1671ad9c805d035f9766cda035c1e7401c1ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0) 2024-03-26T21:55:05.493 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.114589388 +0000 UTC m=+0.017809036 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:55:05.493 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.396636043 +0000 UTC m=+0.299855705 container init 162c104546306cf7a97638f1fc1671ad9c805d035f9766cda035c1e7401c1ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T21:55:05.493 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.399928491 +0000 UTC m=+0.303148148 container start 162c104546306cf7a97638f1fc1671ad9c805d035f9766cda035c1e7401c1ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b) 2024-03-26T21:55:05.493 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.49638629 +0000 UTC m=+0.399605948 container attach 162c104546306cf7a97638f1fc1671ad9c805d035f9766cda035c1e7401c1ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:55:05.741 INFO:journalctl@ceph.osd.5.smithi067.stdout:Mar 26 21:55:05 smithi067 podman[166146]: 2024-03-26 21:55:05.701863676 +0000 UTC m=+0.605083327 container died 162c104546306cf7a97638f1fc1671ad9c805d035f9766cda035c1e7401c1ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:55:06.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T21:55:06.314 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:55:06.314 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T21:55:06.314 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T21:55:06.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.6 2024-03-26T21:55:06.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:06.369+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:06.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:06.369+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:06.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:06.369+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:06.493 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.285+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:06.493 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.285+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:06.493 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.285+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:06.493 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 systemd[1]: Stopping Ceph osd.6 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:06.992 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.662+0000 7fa436515700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:55:06.992 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.662+0000 7fa436515700 -1 osd.6 541 *** Got signal Terminated *** 2024-03-26T21:55:06.992 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:06 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:06.662+0000 7fa436515700 -1 osd.6 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:07.382+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:07.382+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:07.382+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:07.274+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:07.274+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:07.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:07 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:07.274+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:08.347+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:08.347+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:08.347+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:08.241+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:08.241+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:08.241+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:08.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:08 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:08.241+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:43.454740+0000 front 2024-03-26T21:54:43.454588+0000 (oldest deadline 2024-03-26T21:55:08.154614+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:09.283+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:09.283+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:09.283+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:09.283+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:43.454740+0000 front 2024-03-26T21:54:43.454588+0000 (oldest deadline 2024-03-26T21:55:08.154614+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:09.356+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:09.356+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:09.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:09.356+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:09.493 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:09 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:09.356+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:10.381+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:10.381+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:10.381+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:10.381+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:10.256+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:10.256+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:10.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:10.256+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:10.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:10 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:10.256+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:43.454740+0000 front 2024-03-26T21:54:43.454588+0000 (oldest deadline 2024-03-26T21:55:08.154614+0000) 2024-03-26T21:55:11.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:11.339+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:11.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:11.339+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:11.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:11.339+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:11.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:11.339+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:11.742 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:11.301+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:17.151260+0000 front 2024-03-26T21:54:17.151065+0000 (oldest deadline 2024-03-26T21:54:43.051181+0000) 2024-03-26T21:55:11.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:11.301+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:28.853122+0000 front 2024-03-26T21:54:28.853066+0000 (oldest deadline 2024-03-26T21:54:54.152843+0000) 2024-03-26T21:55:11.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:11.301+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.453748+0000 front 2024-03-26T21:54:36.453763+0000 (oldest deadline 2024-03-26T21:55:01.153650+0000) 2024-03-26T21:55:11.743 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:11 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6[49883]: 2024-03-26T21:55:11.301+0000 7fa432b09700 -1 osd.6 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:43.454740+0000 front 2024-03-26T21:54:43.454588+0000 (oldest deadline 2024-03-26T21:55:08.154614+0000) 2024-03-26T21:55:12.492 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:12 smithi067 podman[166251]: 2024-03-26 21:55:12.24396477 +0000 UTC m=+5.692601973 container died e8282dcce933feebf7b37506e8e9e65a2ccdafbc1b1d2ef18308ea9a72826828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T21:55:12.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:12.325+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:12.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:12.325+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:12.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:12.325+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:12.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:12 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:12.325+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:12.884 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:12 smithi067 podman[166251]: 2024-03-26 21:55:12.688110545 +0000 UTC m=+6.136747753 container cleanup e8282dcce933feebf7b37506e8e9e65a2ccdafbc1b1d2ef18308ea9a72826828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T21:55:12.884 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:12 smithi067 bash[166251]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6 2024-03-26T21:55:13.242 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:12 smithi067 podman[166269]: 2024-03-26 21:55:12.889411844 +0000 UTC m=+0.643212437 container remove e8282dcce933feebf7b37506e8e9e65a2ccdafbc1b1d2ef18308ea9a72826828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.211823121 +0000 UTC m=+0.018412763 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.340428652 +0000 UTC m=+0.147018337 container create b76d57ce29f93621d237d6e10c05bc2a80c11657399670075e3064d8a4e80b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:13.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:13.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:13.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:13.538 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:13 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:13.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:13.846 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.542912598 +0000 UTC m=+0.349502238 container init b76d57ce29f93621d237d6e10c05bc2a80c11657399670075e3064d8a4e80b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T21:55:13.846 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.546760182 +0000 UTC m=+0.353349826 container start b76d57ce29f93621d237d6e10c05bc2a80c11657399670075e3064d8a4e80b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T21:55:13.846 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.609293712 +0000 UTC m=+0.415883366 container attach b76d57ce29f93621d237d6e10c05bc2a80c11657399670075e3064d8a4e80b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-26T21:55:14.115 INFO:journalctl@ceph.osd.6.smithi067.stdout:Mar 26 21:55:13 smithi067 podman[166363]: 2024-03-26 21:55:13.851175065 +0000 UTC m=+0.657764716 container died b76d57ce29f93621d237d6e10c05bc2a80c11657399670075e3064d8a4e80b6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T21:55:14.416 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.278+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:14.416 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.278+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:14.416 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.278+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:14.417 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.278+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:14.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T21:55:14.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:55:14.483 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T21:55:14.483 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T21:55:14.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7 2024-03-26T21:55:14.724 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 systemd[1]: Stopping Ceph osd.7 for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:14.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.832+0000 7fe250493700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:55:14.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.832+0000 7fe250493700 -1 osd.7 541 *** Got signal Terminated *** 2024-03-26T21:55:14.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:14 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:14.832+0000 7fe250493700 -1 osd.7 541 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T21:55:15.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:15.241+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:15.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:15.241+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:15.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:15.241+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:15.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:15 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:15.241+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:16.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:16.246+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:16.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:16.246+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:16.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:16.246+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:16.492 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:16 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:16.246+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:17.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:17.271+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:17.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:17.271+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:17.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:17.271+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:17.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:17 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:17.271+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:18.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:18 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:18.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:18.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:18 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:18.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:18.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:18 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:18.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:18.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:18 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:18.316+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:19.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:19.318+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6803 osd.0 since back 2024-03-26T21:54:18.093650+0000 front 2024-03-26T21:54:18.093721+0000 (oldest deadline 2024-03-26T21:54:43.393558+0000) 2024-03-26T21:55:19.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:19.318+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6807 osd.1 since back 2024-03-26T21:54:26.195217+0000 front 2024-03-26T21:54:26.195237+0000 (oldest deadline 2024-03-26T21:54:56.195658+0000) 2024-03-26T21:55:19.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:19.318+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6811 osd.2 since back 2024-03-26T21:54:36.196354+0000 front 2024-03-26T21:54:36.196286+0000 (oldest deadline 2024-03-26T21:55:00.296240+0000) 2024-03-26T21:55:19.742 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:19 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7[117015]: 2024-03-26T21:55:19.318+0000 7fe248a7f700 -1 osd.7 541 heartbeat_check: no reply from 172.21.15.55:6815 osd.3 since back 2024-03-26T21:54:46.097801+0000 front 2024-03-26T21:54:46.097764+0000 (oldest deadline 2024-03-26T21:55:08.397758+0000) 2024-03-26T21:55:20.578 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:20 smithi067 podman[166469]: 2024-03-26 21:55:20.293446298 +0000 UTC m=+5.565864738 container died 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T21:55:20.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:20 smithi067 podman[166469]: 2024-03-26 21:55:20.765933444 +0000 UTC m=+6.038351886 container cleanup 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T21:55:20.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:20 smithi067 bash[166469]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7 2024-03-26T21:55:21.345 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166497]: 2024-03-26 21:55:21.050417171 +0000 UTC m=+0.752696096 container remove 31697ce5adec59cb3a27303f93df079248e18d53c047d7e14320847ac814d634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True) 2024-03-26T21:55:21.674 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.253462453 +0000 UTC m=+0.018050658 image pull a9bdc3eaeb106993bcbbac35d7fe5c7b76c052021676e129f5984aed30aee063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d 2024-03-26T21:55:21.674 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.35962537 +0000 UTC m=+0.124213576 container create 4e90441f7088e961b33e92c8d83f93bc5cf09962d47e1ec1ae93f55d5878a5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T21:55:21.674 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.579005417 +0000 UTC m=+0.343593623 container init 4e90441f7088e961b33e92c8d83f93bc5cf09962d47e1ec1ae93f55d5878a5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-26T21:55:21.674 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.582040654 +0000 UTC m=+0.346628870 container start 4e90441f7088e961b33e92c8d83f93bc5cf09962d47e1ec1ae93f55d5878a5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T21:55:21.674 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.67877938 +0000 UTC m=+0.443367601 container attach 4e90441f7088e961b33e92c8d83f93bc5cf09962d47e1ec1ae93f55d5878a5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T21:55:21.992 INFO:journalctl@ceph.osd.7.smithi067.stdout:Mar 26 21:55:21 smithi067 podman[166590]: 2024-03-26 21:55:21.884981639 +0000 UTC m=+0.649569836 container died 4e90441f7088e961b33e92c8d83f93bc5cf09962d47e1ec1ae93f55d5878a5c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-26T21:55:22.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T21:55:22.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:55:22.613 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T21:55:22.613 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T21:55:22.613 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T21:55:22.913 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:22 smithi055 systemd[1]: Stopping Ceph rgw.foo.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:23.187 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:22 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a[67277]: 2024-03-26T21:55:22.916+0000 7fa4d704b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T21:55:23.187 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:22 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a[67277]: 2024-03-26T21:55:22.916+0000 7fa4e58d1ac0 -1 shutting down 2024-03-26T21:55:33.187 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:32 smithi055 bash[218959]: time="2024-03-26T21:55:32Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T21:55:33.188 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:32 smithi055 podman[218959]: 2024-03-26 21:55:32.932865342 +0000 UTC m=+10.118227991 container died 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-26T21:55:33.568 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 podman[218959]: 2024-03-26 21:55:33.374723654 +0000 UTC m=+10.560086355 container cleanup 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T21:55:33.568 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 bash[218959]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T21:55:33.849 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T21:55:33.873 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 podman[218986]: 2024-03-26 21:55:33.617543225 +0000 UTC m=+0.677890422 container remove 1dca60273bfeb6dec7aca9168e7b3449cfe764712f9dd3aa29cb4c287d9ca8f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-26T21:55:33.873 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T21:55:33.873 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T21:55:33.873 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 systemd[1]: Stopped Ceph rgw.foo.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:33.873 INFO:journalctl@ceph.rgw.foo.a.smithi055.stdout:Mar 26 21:55:33 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 3.829s CPU time. 2024-03-26T21:55:33.887 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:55:33.887 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T21:55:33.887 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T21:55:33.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T21:55:34.171 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:33 smithi067 systemd[1]: Stopping Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:34.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:55:34.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T21:55:34.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T21:55:34.492 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.176Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.180Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.180Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.180Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a[67655]: ts=2024-03-26T21:55:34.181Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T21:55:34.493 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 podman[166695]: 2024-03-26 21:55:34.186262388 +0000 UTC m=+0.096842998 container died ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:34.939 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 podman[166695]: 2024-03-26 21:55:34.71779128 +0000 UTC m=+0.628371943 container cleanup ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:34.939 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 bash[166695]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T21:55:35.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T21:55:35.206 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:34 smithi067 podman[166709]: 2024-03-26 21:55:34.944159291 +0000 UTC m=+0.756106589 container remove ac9cf65bbcf4120a00e01e8b022be10f757cb1a58bcac1bbd64d79a49b89800d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:35.206 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:35 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T21:55:35.207 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:35 smithi067 systemd[1]: Stopped Ceph prometheus.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:35.207 INFO:journalctl@ceph.prometheus.a.smithi067.stdout:Mar 26 21:55:35 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 5.193s CPU time. 2024-03-26T21:55:35.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T21:55:35.219 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T21:55:35.219 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T21:55:35.416 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:55:38.034 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:37 smithi055 systemd[1]: Stopping Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:38.318 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a[78557]: ts=2024-03-26T21:55:38.038Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T21:55:38.318 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 podman[219463]: 2024-03-26 21:55:38.079355659 +0000 UTC m=+0.141777890 container died a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:38.777 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 podman[219463]: 2024-03-26 21:55:38.522262385 +0000 UTC m=+0.584684626 container cleanup a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:38.777 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 bash[219463]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T21:55:39.068 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 podman[219478]: 2024-03-26 21:55:38.781784714 +0000 UTC m=+0.670599589 container remove a86f68806863d5c2bec6946298abd0363d8c984f331066a14af2626c2994b36e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:39.068 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:38 smithi055 podman[219478]: 2024-03-26 21:55:38.849954975 +0000 UTC m=+0.738769850 volume remove 1ceaf5ef1c99cb4180a605e2fcc162f926b0664d8682e887bf45f794f1cf177a 2024-03-26T21:55:39.524 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:39 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T21:55:39.524 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:39 smithi055 systemd[1]: Stopped Ceph alertmanager.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:39.524 INFO:journalctl@ceph.alertmanager.a.smithi055.stdout:Mar 26 21:55:39 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.712s CPU time. 2024-03-26T21:55:39.525 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:39 smithi055 systemd[1]: Stopping Ceph node-exporter.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:39.818 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:39 smithi055 podman[219590]: 2024-03-26 21:55:39.529022291 +0000 UTC m=+0.106044241 container died 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:40.173 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:39 smithi055 podman[219590]: 2024-03-26 21:55:39.985468091 +0000 UTC m=+0.562490030 container cleanup 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:40.173 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:39 smithi055 bash[219590]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T21:55:40.549 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:40 smithi055 podman[219605]: 2024-03-26 21:55:40.178208286 +0000 UTC m=+0.653834280 container remove 821d340909ea09823f8d21a03e349727b1dcedabca67e342b77ac0e27fb3ed49 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T21:55:40.549 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:40 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T21:55:40.549 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:40 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T21:55:40.549 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:40 smithi055 systemd[1]: Stopped Ceph node-exporter.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:40.549 INFO:journalctl@ceph.node-exporter.a.smithi055.stdout:Mar 26 21:55:40 smithi055 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.048s CPU time. 2024-03-26T21:55:41.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T21:55:41.319 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T21:55:43.492 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:43 smithi067 systemd[1]: Stopping Ceph iscsi.iscsi.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:43.992 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:43 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a[60658]: debug Shutdown received 2024-03-26T21:55:53.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:53 smithi067 bash[167102]: time="2024-03-26T21:55:53Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T21:55:53.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:53 smithi067 podman[167102]: 2024-03-26 21:55:53.610860918 +0000 UTC m=+10.139523557 container died 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_BRANCH=HEAD) 2024-03-26T21:55:54.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 podman[167102]: 2024-03-26 21:55:54.088761296 +0000 UTC m=+10.617423975 container cleanup 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-26T21:55:54.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 bash[167102]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T21:55:54.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 podman[167129]: 2024-03-26 21:55:54.22321957 +0000 UTC m=+0.613538901 container remove 0d021b90c8e63b06e66daf03ce5b98e6a12fdb744ef3f935bd3caeac07e1dad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:77a228078e9cbbda8c0ec34daeaf242ac1c029d4330510ad5bc8624b2b23960d, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-26-0925-42fc52b, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-26T21:55:54.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T21:55:54.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T21:55:54.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 systemd[1]: Stopped Ceph iscsi.iscsi.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:54.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi067.stdout:Mar 26 21:55:54 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 3.735s CPU time. 2024-03-26T21:55:56.185 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:55 smithi067 systemd[1]: Stopping Ceph grafana.a for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:56.452 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=server t=2024-03-26T21:55:56.189223296Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-26T21:55:56.452 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=ticker t=2024-03-26T21:55:56.189593617Z level=info msg=stopped last_tick=2024-03-26T21:55:50Z 2024-03-26T21:55:56.453 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a[64334]: logger=tracing t=2024-03-26T21:55:56.189641247Z level=info msg="Closing tracing" 2024-03-26T21:55:56.453 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 podman[167389]: 2024-03-26 21:55:56.209603526 +0000 UTC m=+0.115487426 container died 38bb9105ace78975054c4eafd60dddecd925d2bb2bb0f5d0e5a7cf41e266f7aa (image=quay.io/ceph/grafana:9.4.12, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:55:56.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 podman[167389]: 2024-03-26 21:55:56.639200376 +0000 UTC m=+0.545084282 container cleanup 38bb9105ace78975054c4eafd60dddecd925d2bb2bb0f5d0e5a7cf41e266f7aa (image=quay.io/ceph/grafana:9.4.12, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:55:56.742 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 bash[167389]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a 2024-03-26T21:55:57.233 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:56 smithi067 podman[167404]: 2024-03-26 21:55:56.832187389 +0000 UTC m=+0.620397627 container remove 38bb9105ace78975054c4eafd60dddecd925d2bb2bb0f5d0e5a7cf41e266f7aa (image=quay.io/ceph/grafana:9.4.12, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T21:55:57.233 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:57 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T21:55:57.233 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:57 smithi067 systemd[1]: Stopped Ceph grafana.a for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:57.234 INFO:journalctl@ceph.grafana.a.smithi067.stdout:Mar 26 21:55:57 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@grafana.a.service: Consumed 18.223s CPU time. 2024-03-26T21:55:57.492 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:57 smithi067 systemd[1]: Stopping Ceph node-exporter.b for e5f52b02-ebb6-11ee-95d0-87774f69a715... 2024-03-26T21:55:57.750 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:57 smithi067 podman[167519]: 2024-03-26 21:55:57.498147897 +0000 UTC m=+0.120406949 container died 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:55:58.083 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:57 smithi067 podman[167519]: 2024-03-26 21:55:57.845408509 +0000 UTC m=+0.467667538 container cleanup 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:55:58.083 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:57 smithi067 bash[167519]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T21:55:58.353 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:58 smithi067 podman[167531]: 2024-03-26 21:55:58.088111335 +0000 UTC m=+0.591048568 container remove 89ea3c187a4f2fd11276c612f0c0712e1d76a96c031dcce7b35e1e5f263e8e09 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T21:55:58.353 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:58 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T21:55:58.353 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:58 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T21:55:58.354 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:58 smithi067 systemd[1]: Stopped Ceph node-exporter.b for e5f52b02-ebb6-11ee-95d0-87774f69a715. 2024-03-26T21:55:58.354 INFO:journalctl@ceph.node-exporter.b.smithi067.stdout:Mar 26 21:55:58 smithi067 systemd[1]: ceph-e5f52b02-ebb6-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 4.689s CPU time. 2024-03-26T21:55:58.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:55:58.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T21:55:58.954 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T21:55:58.956 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi055/crash 2024-03-26T21:55:58.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash -- . 2024-03-26T21:55:59.014 INFO:teuthology.orchestra.run.smithi055.stderr:tar: /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T21:55:59.014 INFO:teuthology.orchestra.run.smithi055.stderr:tar: Error is not recoverable: exiting now 2024-03-26T21:55:59.017 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi067/crash 2024-03-26T21:55:59.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash -- . 2024-03-26T21:55:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T21:55:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-03-26T21:55:59.065 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T21:55:59.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T21:55:59.146 INFO:teuthology.orchestra.run.smithi055.stdout:2024-03-26T21:30:30.725236+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:55:59.150 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T21:55:59.150 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[SEC\]' /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T21:55:59.182 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]' /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T21:55:59.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[WRN\]' /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T21:55:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:2024-03-26T21:30:30.725236+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T21:55:59.325 INFO:tasks.cephadm:Compressing logs... 2024-03-26T21:55:59.326 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T21:55:59.369 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T21:55:59.419 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-26T21:55:59.420 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T21:55:59.422 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-26T21:55:59.422 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T21:55:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T21:55:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log 2024-03-26T21:55:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T21:55:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T21:55:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/objectstore_tool.2.log 2024-03-26T21:55:59.429 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T21:55:59.429 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T21:55:59.429 INFO:teuthology.orchestra.run.smithi055.stderr:gzip/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T21:55:59.430 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T21:55:59.430 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T21:55:59.430 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.1.log: 57.6% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T21:55:59.432 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T21:55:59.433 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T21:55:59.444 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T21:55:59.500 INFO:teuthology.orchestra.run.smithi055.stderr: 89.2% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T21:55:59.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.b.log: /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/objectstore_tool.2.log: 67.4% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T21:55:59.552 INFO:teuthology.orchestra.run.smithi067.stderr: 81.6% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T21:55:59.569 INFO:teuthology.orchestra.run.smithi067.stderr: 90.1% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T21:55:59.583 INFO:teuthology.orchestra.run.smithi067.stderr: 92.3% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T21:55:59.631 INFO:teuthology.orchestra.run.smithi067.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T21:55:59.649 INFO:teuthology.orchestra.run.smithi055.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T21:55:59.682 INFO:teuthology.orchestra.run.smithi055.stderr: 94.0% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T21:55:59.686 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T21:55:59.923 INFO:teuthology.orchestra.run.smithi055.stderr: 91.2% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T21:56:00.362 INFO:teuthology.orchestra.run.smithi067.stderr: 94.7% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/objectstore_tool.2.log.gz 2024-03-26T21:56:00.902 INFO:teuthology.orchestra.run.smithi055.stderr: 89.6% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T21:56:01.087 INFO:teuthology.orchestra.run.smithi067.stderr: 85.5% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T21:56:03.231 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T21:56:15.218 INFO:teuthology.orchestra.run.smithi055.stderr: 91.1% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T21:56:43.695 INFO:teuthology.orchestra.run.smithi055.stderr: 90.3% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T21:59:49.376 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T21:59:49.415 INFO:teuthology.orchestra.run.smithi067.stderr: 92.4% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T22:04:21.914 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T22:05:05.473 INFO:teuthology.orchestra.run.smithi067.stderr: 92.4% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T22:05:05.475 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-26T22:05:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:real 9m6.088s 2024-03-26T22:05:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:user 6m19.142s 2024-03-26T22:05:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m16.018s 2024-03-26T22:16:30.368 INFO:teuthology.orchestra.run.smithi055.stderr: 92.2% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T22:18:36.266 INFO:teuthology.orchestra.run.smithi055.stderr: 92.5% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T22:19:20.834 INFO:teuthology.orchestra.run.smithi055.stderr: 92.6% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T22:19:55.238 INFO:teuthology.orchestra.run.smithi055.stderr: 92.1% -- replaced with /var/log/ceph/e5f52b02-ebb6-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T22:19:55.240 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-26T22:19:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:real 23m55.853s 2024-03-26T22:19:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:user 9m5.322s 2024-03-26T22:19:55.240 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m24.214s 2024-03-26T22:19:55.240 INFO:tasks.cephadm:Archiving logs... 2024-03-26T22:19:55.241 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi055/log 2024-03-26T22:19:55.242 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:21:18.928 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi067/log 2024-03-26T22:21:18.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T22:22:16.795 INFO:tasks.cephadm:Removing cluster... 2024-03-26T22:22:16.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 --force 2024-03-26T22:22:17.031 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T22:22:17.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e5f52b02-ebb6-11ee-95d0-87774f69a715 --force 2024-03-26T22:22:18.247 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: e5f52b02-ebb6-11ee-95d0-87774f69a715 2024-03-26T22:22:19.015 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T22:22:19.015 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T22:22:19.034 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T22:22:19.049 INFO:tasks.cephadm:Teardown complete 2024-03-26T22:22:19.050 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T22:22:19.062 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T22:22:19.062 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:22:19.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T22:22:19.201 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T22:22:19.215 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T22:22:19.216 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:22:19.219 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T22:22:19.233 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:22:19.234 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-26T22:22:19.265 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:22:19.266 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-26T22:22:19.266 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +395us[ +395us] +/- 89ms 2024-03-26T22:22:19.266 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -852us[ -862us] +/- 69ms 2024-03-26T22:22:19.266 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +737us[ +728us] +/- 89ms 2024-03-26T22:22:19.266 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:22:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T22:22:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-26T22:22:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +387us[ +387us] +/- 89ms 2024-03-26T22:22:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -874us[ -874us] +/- 69ms 2024-03-26T22:22:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +732us[ +732us] +/- 89ms 2024-03-26T22:22:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T22:22:19.286 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T22:22:19.300 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T22:22:19.301 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T22:22:19.324 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:22:19.424 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T22:22:19.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T22:22:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:22:19.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T22:22:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:22:19.553 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T22:22:19.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T22:22:19.580 INFO:teuthology.task.internal:Duration was 4142.949292 seconds 2024-03-26T22:22:19.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T22:22:19.597 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T22:22:19.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:22:19.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T22:22:19.670 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:22:19.695 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T22:22:20.148 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T22:22:20.148 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-03-26T22:22:20.148 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:22:20.171 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-03-26T22:22:20.171 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T22:22:20.208 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T22:22:20.208 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:22:20.214 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T22:22:20.277 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T22:22:20.277 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:22:20.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T22:22:21.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T22:22:21.911 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T22:22:21.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:22:21.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T22:22:21.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T22:22:22.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:22:22.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:22:22.036 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-03-26T22:22:22.080 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-03-26T22:22:22.120 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:22:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:22:22.133 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T22:22:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T22:22:22.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T22:22:22.159 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T22:22:22.160 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi055 2024-03-26T22:22:22.160 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:22:22.241 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129/remote/smithi067 2024-03-26T22:22:22.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T22:22:22.285 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T22:22:22.285 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:22:22.288 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T22:22:22.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T22:22:22.347 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T22:22:22.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T22:22:22.367 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T22:22:22.367 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:22:22.369 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T22:22:22.381 INFO:teuthology.orchestra.run.smithi055.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:22 /home/ubuntu/cephtest 2024-03-26T22:22:22.389 INFO:teuthology.orchestra.run.smithi067.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 22:22 /home/ubuntu/cephtest 2024-03-26T22:22:22.391 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T22:22:22.402 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T22:22:22.438 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T22:22:22.466 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-26T22:22:22.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:22:22.488 DEBUG:teuthology.nuke:shortname: smithi067 2024-03-26T22:22:22.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T22:22:22.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:04:21.293410', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLl8JtZZia57JYJs08MTbvDulZpYGj2y1JE7EAG8GqkYw4GqBTxLBvOESKK+7Nsc2h2cBCEVEUnctAIu9B1fPCI='} 2024-03-26T22:22:22.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-26_20:31:35-orch:cephadm-wip-guits-testing-2024-03-26-0925-distro-default-smithi/7624129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 21:04:21.291743', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPDWtrlkI5YzzjVmQcYHd88AMtVeANnyY2U5TB9zHZmkKGfAvcY6gUqtn3RyRUHvalZkcEGAT9kD7bFchZPqsjQ='} 2024-03-26T22:22:22.539 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-26T22:22:22.539 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:22:22.564 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-26T22:22:22.565 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:22.579 INFO:teuthology.orchestra.console.smithi067:Power off 2024-03-26T22:22:22.580 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T22:22:22.589 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-26T22:22:22.611 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-03-26T22:22:22.611 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:22.635 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-26T22:22:26.591 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:26.637 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:26.702 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-26T22:22:26.746 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-26T22:22:30.706 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:30.748 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:30.816 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-26T22:22:30.858 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-03-26T22:22:30.958 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-03-26T22:22:34.818 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T22:22:34.928 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-26T22:22:35.028 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-26T22:22:35.132 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 4142.949292182922 failure_reason: '"2024-03-26T21:30:30.725236+0000 mon.a (mon.0) 835 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-03-26T22:22:35.132 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T22:22:35.216 INFO:teuthology.run:FAIL